[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:46:42 dist-test-slave-7lbf krb5kdc[4219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:46:42 dist-test-slave-7lbf krb5kdc[4219](info): set up 2 sockets
Aug 05 22:46:42 dist-test-slave-7lbf krb5kdc[4219](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260805 22:46:43.703056 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:46:43 dist-test-slave-7lbf krb5kdc[4219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:46:43.922715 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:40215
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:40215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:44.501719 4235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:44.502224 4235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:44.502372 4235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:44.510151 4235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:46:44.510298 4235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:44.510382 4235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:44.510459 4235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:46:44.510522 4235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:46:44.521735 4235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:40215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:40215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:44.523980 4235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:44.526180 4235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:44.555905 4240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:44.555680 4241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:44.556205 4244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:44.557554 4235 server_base.cc:1034] running on GCE node
I20260805 22:46:44.558892 4235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:44.561422 4235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:44.562765 4235 hybrid_clock.cc:648] HybridClock initialized: now 1785970004562684 us; error 110 us; skew 500 ppm
I20260805 22:46:44.563426 4235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:44 dist-test-slave-7lbf krb5kdc[4219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970004, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:44.567205 4235 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:46:44.569943 4235 webserver.cc:460] Webserver started at http://127.4.28.62:45111/ using document root <none> and password file <none>
I20260805 22:46:44.571177 4235 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:44.571317 4235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:44.571861 4235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:44.576368 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "d554cdf346ff4d9b80a733269155074a"
format_stamp: "Formatted at 2026-08-05 22:46:44 on dist-test-slave-7lbf"
server_key: "7c6a65d472e64095283150465e6444a8"
server_key_iv: "c131cf4d847f7023f96ceb1e6abb7172"
server_key_version: "encryptionkey@0"
I20260805 22:46:44.577292 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "d554cdf346ff4d9b80a733269155074a"
format_stamp: "Formatted at 2026-08-05 22:46:44 on dist-test-slave-7lbf"
server_key: "7c6a65d472e64095283150465e6444a8"
server_key_iv: "c131cf4d847f7023f96ceb1e6abb7172"
server_key_version: "encryptionkey@0"
I20260805 22:46:44.586133 4235 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260805 22:46:44.592299 4251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:44.594676 4235 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:46:44.594987 4235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "d554cdf346ff4d9b80a733269155074a"
format_stamp: "Formatted at 2026-08-05 22:46:44 on dist-test-slave-7lbf"
server_key: "7c6a65d472e64095283150465e6444a8"
server_key_iv: "c131cf4d847f7023f96ceb1e6abb7172"
server_key_version: "encryptionkey@0"
I20260805 22:46:44.595225 4235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:44.611891 4235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:44.618083 4235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:44.618481 4235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:44.642094 4235 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:40215
I20260805 22:46:44.642112 4320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:40215 every 8 connection(s)
I20260805 22:46:44.644801 4235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:46:44.652350 4321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:44.654621 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4235
I20260805 22:46:44.654996 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:46:44.657282 4208 external_mini_cluster.cc:1428] Setting key 56404ffe58cc6abf021b7a6c744e6e82
I20260805 22:46:44.671501 4321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a: Bootstrap starting.
I20260805 22:46:44.679409 4321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:44.681589 4321 log.cc:824] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a: Log is configured to *not* fsync() on all Append() calls
I20260805 22:46:44.687178 4321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a: No bootstrap required, opened a new log
Aug 05 22:46:44 dist-test-slave-7lbf krb5kdc[4219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:44.699672 4321 raft_consensus.cc:348] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554cdf346ff4d9b80a733269155074a" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40215 } }
I20260805 22:46:44.700378 4321 raft_consensus.cc:374] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:44.700661 4321 raft_consensus.cc:729] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d554cdf346ff4d9b80a733269155074a, State: Initialized, Role: FOLLOWER
I20260805 22:46:44.702113 4321 consensus_queue.cc:260] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554cdf346ff4d9b80a733269155074a" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40215 } }
I20260805 22:46:44.702490 4321 raft_consensus.cc:388] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:46:44.702756 4321 raft_consensus.cc:482] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:46:44.703001 4321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:44.705703 4321 raft_consensus.cc:504] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554cdf346ff4d9b80a733269155074a" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40215 } }
I20260805 22:46:44.707018 4321 leader_election.cc:302] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d554cdf346ff4d9b80a733269155074a; no voters:
I20260805 22:46:44.708130 4321 leader_election.cc:288] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:46:44.708447 4328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:46:44.709985 4328 raft_consensus.cc:686] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [term 1 LEADER]: Becoming Leader. State: Replica: d554cdf346ff4d9b80a733269155074a, State: Running, Role: LEADER
I20260805 22:46:44.710943 4328 consensus_queue.cc:237] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554cdf346ff4d9b80a733269155074a" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40215 } }
I20260805 22:46:44.712265 4321 sys_catalog.cc:564] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:46:44.712888 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:44.664286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.4.28.62:40215)
0805 22:46:44.665767 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:44.665787 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:44.668159 (+ 2372us) server_negotiation.cc:407] Connection header received
0805 22:46:44.682010 (+ 13851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:44.682041 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:44.682931 (+ 890us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:44.683551 (+ 620us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:44.688456 (+ 4905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:44.690055 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:44.692625 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:44.693434 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:44.693700 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:44.701678 (+ 7978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:44.701763 (+ 85us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:44.701805 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:44.701871 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:44.705197 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:44.706482 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:44.706491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:44.706497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:44.706766 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:44.707438 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:44.707442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:44.707444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:44.708444 (+ 1000us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:44.708866 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:44.710512 (+ 1646us) server_negotiation.cc:299] Negotiation successful
0805 22:46:44.711005 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":425,"threads_started":1}
I20260805 22:46:44.717404 4329 sys_catalog.cc:455] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d554cdf346ff4d9b80a733269155074a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554cdf346ff4d9b80a733269155074a" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40215 } } }
I20260805 22:46:44.718307 4329 sys_catalog.cc:458] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [sys.catalog]: This master's current role is: LEADER
I20260805 22:46:44.718439 4330 sys_catalog.cc:455] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d554cdf346ff4d9b80a733269155074a. Latest consensus state: current_term: 1 leader_uuid: "d554cdf346ff4d9b80a733269155074a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554cdf346ff4d9b80a733269155074a" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40215 } } }
I20260805 22:46:44.719192 4330 sys_catalog.cc:458] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a [sys.catalog]: This master's current role is: LEADER
I20260805 22:46:44.724951 4338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:46:44.734297 4338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:46:44.749491 4338 catalog_manager.cc:1347] Generated new cluster ID: cef10de184a04487bf17a50c55c3c9e9
I20260805 22:46:44.749665 4338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:46:44.778120 4338 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:46:44.779690 4338 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:46:44.786300 4338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d554cdf346ff4d9b80a733269155074a: Generated new TSK 0
I20260805 22:46:44.787834 4338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:46:44.850710 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40215
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:45.299973 4354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:45.300328 4354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:45.300458 4354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:45.307979 4354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:45.308104 4354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:45.308234 4354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:46:45.318617 4354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:45.320009 4354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:45.321465 4354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:45.336680 4359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:45.336720 4361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:45.336635 4364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:45.337690 4354 server_base.cc:1034] running on GCE node
I20260805 22:46:45.338407 4354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:45.339563 4354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:45.340811 4354 hybrid_clock.cc:648] HybridClock initialized: now 1785970005340752 us; error 54 us; skew 500 ppm
I20260805 22:46:45.341166 4354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:45 dist-test-slave-7lbf krb5kdc[4219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970005, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:45.344659 4354 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:46:45.346477 4354 webserver.cc:460] Webserver started at http://127.4.28.1:41529/ using document root <none> and password file <none>
I20260805 22:46:45.347249 4354 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:45.347362 4354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:45.347652 4354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:45.350939 4354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "f2841afe953142c18d9b73b30a0440f1"
format_stamp: "Formatted at 2026-08-05 22:46:45 on dist-test-slave-7lbf"
server_key: "31b752e2f407757cdea73ebb69d5492e"
server_key_iv: "14798037382904a5be7731c7a83be1d3"
server_key_version: "encryptionkey@0"
I20260805 22:46:45.351735 4354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "f2841afe953142c18d9b73b30a0440f1"
format_stamp: "Formatted at 2026-08-05 22:46:45 on dist-test-slave-7lbf"
server_key: "31b752e2f407757cdea73ebb69d5492e"
server_key_iv: "14798037382904a5be7731c7a83be1d3"
server_key_version: "encryptionkey@0"
I20260805 22:46:45.357414 4354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260805 22:46:45.361227 4379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:45.362723 4354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:46:45.362884 4354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "f2841afe953142c18d9b73b30a0440f1"
format_stamp: "Formatted at 2026-08-05 22:46:45 on dist-test-slave-7lbf"
server_key: "31b752e2f407757cdea73ebb69d5492e"
server_key_iv: "14798037382904a5be7731c7a83be1d3"
server_key_version: "encryptionkey@0"
I20260805 22:46:45.363155 4354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:45.382568 4354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:45.387221 4354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:45.387499 4354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:45.388705 4354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:46:45.390647 4354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:46:45.390725 4354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:45.390897 4354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:46:45.390959 4354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:45.426286 4354 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:35545
I20260805 22:46:45.426352 4534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:35545 every 8 connection(s)
I20260805 22:46:45.427680 4354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:46:45.437999 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4354
I20260805 22:46:45.438275 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:46:45.438943 4208 external_mini_cluster.cc:1428] Setting key 1b9d78c8de2d5f56f48d149143ff6304
Aug 05 22:46:45 dist-test-slave-7lbf krb5kdc[4219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970005, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:45.451833 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:45.431244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:37811 (local address 127.4.28.62:40215)
0805 22:46:45.431708 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:45.431713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:45.432823 (+ 1110us) server_negotiation.cc:407] Connection header received
0805 22:46:45.434979 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:45.434985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:45.435110 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:45.435218 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:45.437066 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:45.438042 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:45.439946 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:45.440521 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:45.440635 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:45.445211 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:45.445218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:45.445221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:45.445262 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:45.448157 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:45.449124 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:45.449131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:45.449134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:45.449273 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:45.449787 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:45.449791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:45.449793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:45.450120 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:45.450266 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:45.451170 (+ 904us) server_negotiation.cc:299] Negotiation successful
0805 22:46:45.451439 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260805 22:46:45.452915 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:45.431760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40215 (local address 127.4.28.1:37811)
0805 22:46:45.432627 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:45.432672 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:45.434630 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:45.435538 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:45.435548 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:45.436163 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:45.436840 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:45.436860 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:45.438282 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:45.438287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:45.439741 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:45.439757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:45.440803 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:45.440812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:45.441138 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:45.441893 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:45.441935 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:45.444898 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:45.448343 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:45.448354 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:45.448375 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:45.448978 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:45.449396 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:45.449400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:45.449403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:45.449659 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:45.450262 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:45.450271 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:45.450678 (+ 407us) client_negotiation.cc:769] Sending connection context
0805 22:46:45.451040 (+ 362us) client_negotiation.cc:240] Negotiation successful
0805 22:46:45.451653 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":372,"threads_started":1}
I20260805 22:46:45.454816 4536 heartbeater.cc:347] Connected to a master server at 127.4.28.62:40215
I20260805 22:46:45.455348 4536 heartbeater.cc:464] Registering TS with master...
I20260805 22:46:45.456419 4536 heartbeater.cc:511] Master 127.4.28.62:40215 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
I20260805 22:46:45.461014 4274 ts_manager.cc:194] Registered new tserver with Master: f2841afe953142c18d9b73b30a0440f1 (127.4.28.1:35545)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
I20260805 22:46:45.463343 4274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:37811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:46:45.488813 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40215
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:45.847995 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:45.848344 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:45.848485 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:45.856031 4544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:45.856173 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:45.856338 4544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:46:45.866878 4544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:45.868391 4544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:45.869980 4544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:45.884179 4550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:45.884145 4552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:45.884431 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:45.885394 4544 server_base.cc:1034] running on GCE node
I20260805 22:46:45.886231 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:45.887326 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:45.888567 4544 hybrid_clock.cc:648] HybridClock initialized: now 1785970005888495 us; error 59 us; skew 500 ppm
I20260805 22:46:45.889019 4544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:45 dist-test-slave-7lbf krb5kdc[4219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970005, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:45.893081 4544 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:46:45.894750 4544 webserver.cc:460] Webserver started at http://127.4.28.2:39931/ using document root <none> and password file <none>
I20260805 22:46:45.895499 4544 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:45.895613 4544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:45.896086 4544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:45.899379 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "3b0c126c19a4423ab29cb8530b01a9ed"
format_stamp: "Formatted at 2026-08-05 22:46:45 on dist-test-slave-7lbf"
server_key: "11b1915bd1202ed43a59149bae590022"
server_key_iv: "13d42829cba22bfcbfbef35fc4cf107c"
server_key_version: "encryptionkey@0"
I20260805 22:46:45.900317 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "3b0c126c19a4423ab29cb8530b01a9ed"
format_stamp: "Formatted at 2026-08-05 22:46:45 on dist-test-slave-7lbf"
server_key: "11b1915bd1202ed43a59149bae590022"
server_key_iv: "13d42829cba22bfcbfbef35fc4cf107c"
server_key_version: "encryptionkey@0"
I20260805 22:46:45.906190 4544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260805 22:46:45.910162 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:45.911697 4544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:46:45.911901 4544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "3b0c126c19a4423ab29cb8530b01a9ed"
format_stamp: "Formatted at 2026-08-05 22:46:45 on dist-test-slave-7lbf"
server_key: "11b1915bd1202ed43a59149bae590022"
server_key_iv: "13d42829cba22bfcbfbef35fc4cf107c"
server_key_version: "encryptionkey@0"
I20260805 22:46:45.912123 4544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:45.944247 4544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:45.948772 4544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:45.949076 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:45.950198 4544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:46:45.951718 4544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:46:45.951808 4544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:45.951951 4544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:46:45.952018 4544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:45.985329 4544 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:36053
I20260805 22:46:45.985373 4721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:36053 every 8 connection(s)
I20260805 22:46:45.986783 4544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:46:45.991915 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4544
I20260805 22:46:45.992100 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:46:45.992569 4208 external_mini_cluster.cc:1428] Setting key 3b9bbb71fb0a04fe10733eb184732a08
Aug 05 22:46:45 dist-test-slave-7lbf krb5kdc[4219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970005, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:46.010552 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:45.989743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:47305 (local address 127.4.28.62:40215)
0805 22:46:45.990260 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:45.990265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:45.991482 (+ 1217us) server_negotiation.cc:407] Connection header received
0805 22:46:45.993723 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:45.993730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:45.993834 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:45.993937 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:45.995915 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:45.996676 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:45.998466 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:45.998929 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:45.999054 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:46.003313 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:46.003321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:46.003326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:46.003371 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:46.006770 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.007724 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.007731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.007734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.007913 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.008400 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.008405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.008408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.008785 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:46.008945 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:46.009827 (+ 882us) server_negotiation.cc:299] Negotiation successful
0805 22:46:46.010125 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":259,"threads_started":1}
I20260805 22:46:46.011525 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:45.990399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40215 (local address 127.4.28.2:47305)
0805 22:46:45.991245 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:45.991289 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:45.993344 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:45.994253 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:45.994263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:45.995030 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:45.995714 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:45.995736 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:45.996859 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:45.996864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:45.998287 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:45.998313 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:45.999213 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:45.999222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:45.999515 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:46.000222 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:46.000248 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:46.002954 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:46.006992 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:46.007002 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:46.007021 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:46.007556 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:46.008030 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:46.008034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:46.008037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:46.008259 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:46.008937 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:46.008945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:46.009351 (+ 406us) client_negotiation.cc:769] Sending connection context
0805 22:46:46.009655 (+ 304us) client_negotiation.cc:240] Negotiation successful
0805 22:46:46.010179 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":364,"threads_started":1}
I20260805 22:46:46.013309 4723 heartbeater.cc:347] Connected to a master server at 127.4.28.62:40215
I20260805 22:46:46.013679 4723 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
I20260805 22:46:46.014566 4723 heartbeater.cc:511] Master 127.4.28.62:40215 requested a full tablet report, sending...
I20260805 22:46:46.016275 4274 ts_manager.cc:194] Registered new tserver with Master: 3b0c126c19a4423ab29cb8530b01a9ed (127.4.28.2:36053)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
I20260805 22:46:46.017292 4274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:47305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:46:46.042536 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40215
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:46.395298 4733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:46.395664 4733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:46.395813 4733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:46.403736 4733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:46.403901 4733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:46.404078 4733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:46:46.415534 4733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40215
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:46.417074 4733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:46.418588 4733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:46.432955 4740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:46.433020 4741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:46.433319 4743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:46.434603 4733 server_base.cc:1034] running on GCE node
I20260805 22:46:46.435395 4733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:46.436656 4733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:46.437947 4733 hybrid_clock.cc:648] HybridClock initialized: now 1785970006437884 us; error 42 us; skew 500 ppm
I20260805 22:46:46.438316 4733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:46 dist-test-slave-7lbf krb5kdc[4219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970006, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:46.441723 4733 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:46:46.443382 4733 webserver.cc:460] Webserver started at http://127.4.28.3:42235/ using document root <none> and password file <none>
I20260805 22:46:46.444121 4733 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:46.444233 4733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:46.444525 4733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:46.447767 4733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "c435e4a7b3894fada5b9863b5e846c83"
format_stamp: "Formatted at 2026-08-05 22:46:46 on dist-test-slave-7lbf"
server_key: "1b8d895fd3763a0d4cca1d9a8d5c5a89"
server_key_iv: "1ba19961e492eeed427fd8f3f75bc1a9"
server_key_version: "encryptionkey@0"
I20260805 22:46:46.448596 4733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "c435e4a7b3894fada5b9863b5e846c83"
format_stamp: "Formatted at 2026-08-05 22:46:46 on dist-test-slave-7lbf"
server_key: "1b8d895fd3763a0d4cca1d9a8d5c5a89"
server_key_iv: "1ba19961e492eeed427fd8f3f75bc1a9"
server_key_version: "encryptionkey@0"
I20260805 22:46:46.454727 4733 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260805 22:46:46.458850 4755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:46.460489 4733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:46:46.460691 4733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "c435e4a7b3894fada5b9863b5e846c83"
format_stamp: "Formatted at 2026-08-05 22:46:46 on dist-test-slave-7lbf"
server_key: "1b8d895fd3763a0d4cca1d9a8d5c5a89"
server_key_iv: "1ba19961e492eeed427fd8f3f75bc1a9"
server_key_version: "encryptionkey@0"
I20260805 22:46:46.460914 4733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:46.468079 4536 heartbeater.cc:503] Master 127.4.28.62:40215 was elected leader, sending a full tablet report...
I20260805 22:46:46.496688 4733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:46.501412 4733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:46.501734 4733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:46.502790 4733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:46:46.504210 4733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:46:46.504287 4733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:46.504381 4733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:46:46.504432 4733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:46.538079 4733 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:36639
I20260805 22:46:46.538153 4924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:36639 every 8 connection(s)
I20260805 22:46:46.539413 4733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:46:46.545145 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4733
I20260805 22:46:46.545329 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:46:46.545815 4208 external_mini_cluster.cc:1428] Setting key 31a7a375f95c102766e037b0a77670a3
Aug 05 22:46:46 dist-test-slave-7lbf krb5kdc[4219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970006, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:46.562208 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:46.542349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:41103 (local address 127.4.28.62:40215)
0805 22:46:46.542867 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:46.542871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:46.544004 (+ 1133us) server_negotiation.cc:407] Connection header received
0805 22:46:46.546188 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:46.546195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:46.546306 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:46.546426 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:46.548461 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.549471 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.551351 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.551835 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.551976 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:46.556063 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:46.556069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:46.556072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:46.556116 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:46.558794 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.559620 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.559625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.559628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.559774 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.560242 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.560246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.560248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.560562 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:46.560715 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:46.561498 (+ 783us) server_negotiation.cc:299] Negotiation successful
0805 22:46:46.561789 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":244,"threads_started":1}
I20260805 22:46:46.563437 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:46.542943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40215 (local address 127.4.28.3:41103)
0805 22:46:46.543752 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:46.543847 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:46.545846 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:46.546754 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:46.546769 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:46.547463 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:46.548260 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:46.548281 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.549673 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.549677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:46.551163 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:46.551180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.552110 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.552119 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:46.552387 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:46.553156 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:46.553184 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:46.555753 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:46.558970 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:46.558978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:46.558994 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:46.559477 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:46.559892 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:46.559896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:46.559898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:46.560118 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:46.560706 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:46.560716 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:46.561108 (+ 392us) client_negotiation.cc:769] Sending connection context
0805 22:46:46.561382 (+ 274us) client_negotiation.cc:240] Negotiation successful
0805 22:46:46.562026 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":313,"threads_started":1}
I20260805 22:46:46.565001 4926 heartbeater.cc:347] Connected to a master server at 127.4.28.62:40215
I20260805 22:46:46.565326 4926 heartbeater.cc:464] Registering TS with master...
I20260805 22:46:46.566329 4926 heartbeater.cc:511] Master 127.4.28.62:40215 requested a full tablet report, sending...
I20260805 22:46:46.568032 4274 ts_manager.cc:194] Registered new tserver with Master: c435e4a7b3894fada5b9863b5e846c83 (127.4.28.3:36639)
I20260805 22:46:46.568912 4274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:41103
I20260805 22:46:46.576182 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 22:46:46 dist-test-slave-7lbf krb5kdc[4219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 05 22:46:46 dist-test-slave-7lbf krb5kdc[4219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:46:46.632472 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:46.613069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.4.28.1:35545)
0805 22:46:46.613908 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:46.613918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:46.613955 (+ 37us) server_negotiation.cc:407] Connection header received
0805 22:46:46.614049 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:46.614059 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:46.614378 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:46.614571 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:46.616491 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.618306 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.620370 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.621139 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.621257 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:46.626192 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:46.626216 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:46.626241 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:46.626306 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:46.629233 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.630153 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.630162 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.630164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.630318 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.630730 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.630733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.630735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.631168 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:46.631421 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:46.631779 (+ 358us) server_negotiation.cc:299] Negotiation successful
0805 22:46:46.632105 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":358,"threads_started":1}
W20260805 22:46:46.634065 4416 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41840
Aug 05 22:46:46 dist-test-slave-7lbf krb5kdc[4219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 22:46:46 dist-test-slave-7lbf krb5kdc[4219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:46:46.688900 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:46.674007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.4.28.1:35545)
0805 22:46:46.674248 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:46.674256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:46.674598 (+ 342us) server_negotiation.cc:407] Connection header received
0805 22:46:46.674790 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:46.674794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:46.674880 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:46.674981 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:46.676450 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.677176 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.678525 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.678986 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.679111 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:46.683450 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:46.683456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:46.683460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:46.683505 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:46.686138 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.686909 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.686914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.686916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.687077 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.687474 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.687478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.687480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.687762 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:46.688011 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:46.688332 (+ 321us) server_negotiation.cc:299] Negotiation successful
0805 22:46:46.688569 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260805 22:46:46.691262 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4354
I20260805 22:46:46.705452 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:46.692317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.4.28.1:35545)
0805 22:46:46.692543 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:46.692550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:46.692926 (+ 376us) server_negotiation.cc:407] Connection header received
0805 22:46:46.693098 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:46.693103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:46.693167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:46.693258 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:46.694599 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.695352 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.696691 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.697152 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.697273 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:46.699880 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:46.699886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:46.699889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:46.699928 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:46.702701 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.703441 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.703446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.703448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.703601 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.703987 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.703991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.704000 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.704280 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:46.704524 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:46.704850 (+ 326us) server_negotiation.cc:299] Negotiation successful
0805 22:46:46.705090 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:46:46.706363 4526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:46:46.797873 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4354
I20260805 22:46:46.821768 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4544
Aug 05 22:46:46 dist-test-slave-7lbf krb5kdc[4219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:46:46.840505 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:46.822798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.4.28.2:36053)
0805 22:46:46.823373 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:46.823388 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:46.823424 (+ 36us) server_negotiation.cc:407] Connection header received
0805 22:46:46.823507 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:46.823514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:46.823746 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:46.823936 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:46.825389 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.826533 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.827900 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.828530 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.828673 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:46.833547 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:46.833571 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:46.833596 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:46.833655 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:46.837273 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.838182 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.838191 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.838193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.838334 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.838780 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.838783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.838785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.839193 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:46.839446 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:46.839824 (+ 378us) server_negotiation.cc:299] Negotiation successful
0805 22:46:46.840082 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":283,"threads_started":1}
I20260805 22:46:46.842118 4717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:46:46.926291 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4544
I20260805 22:46:46.946553 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4733
Aug 05 22:46:46 dist-test-slave-7lbf krb5kdc[4219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:46:46.964058 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:46.947694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.4.28.3:36639)
0805 22:46:46.948395 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:46.948404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:46.948437 (+ 33us) server_negotiation.cc:407] Connection header received
0805 22:46:46.948502 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:46.948509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:46.948732 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:46.948873 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:46.950406 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.951517 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.952935 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:46.953730 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:46.953832 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:46.958206 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:46.958222 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:46.958241 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:46.958292 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:46.961273 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.962102 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.962110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.962113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.962250 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:46.962565 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:46.962568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:46.962570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:46.962958 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:46.963202 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:46.963422 (+ 220us) server_negotiation.cc:299] Negotiation successful
0805 22:46:46.963644 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":360,"threads_started":1}
I20260805 22:46:46.965641 4920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:46:47.049072 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4733
I20260805 22:46:47.074167 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4235
I20260805 22:46:47.075446 4315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:46:47.179502 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4235
2026-08-05T22:46:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4688 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:46:47 dist-test-slave-7lbf krb5kdc[4957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:46:47 dist-test-slave-7lbf krb5kdc[4957](info): set up 2 sockets
Aug 05 22:46:47 dist-test-slave-7lbf krb5kdc[4957](info): commencing operation
krb5kdc: starting...
W20260805 22:46:48.355667 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:46:48 dist-test-slave-7lbf krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:46:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:46:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:46:48.581161 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:41539
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:41539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:48.934654 4974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:48.935010 4974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:48.935151 4974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:48.942580 4974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:46:48.942703 4974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:48.942776 4974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:48.942867 4974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:46:48.942924 4974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:46:48.953204 4974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:41539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:41539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:48.954664 4974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:48.956202 4974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:48.968689 4981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:48.968732 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:48.968732 4980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:48.969501 4974 server_base.cc:1034] running on GCE node
I20260805 22:46:48.970377 4974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:48.971901 4974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:48.973127 4974 hybrid_clock.cc:648] HybridClock initialized: now 1785970008973066 us; error 52 us; skew 500 ppm
I20260805 22:46:48.973542 4974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:48 dist-test-slave-7lbf krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:48.981544 4974 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:46:48.983317 4974 webserver.cc:460] Webserver started at http://127.4.28.62:41619/ using document root <none> and password file <none>
I20260805 22:46:48.984153 4974 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:48.984292 4974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:48.984635 4974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:48.987967 4974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "6ab2e501c77b48a68451ed064e928ed7"
format_stamp: "Formatted at 2026-08-05 22:46:48 on dist-test-slave-7lbf"
server_key: "6b1e1f3fb03141491c9e8cfb73f48754"
server_key_iv: "81280abba29ac8d5b1b0dfb3fdfe9bcb"
server_key_version: "encryptionkey@0"
I20260805 22:46:48.988791 4974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "6ab2e501c77b48a68451ed064e928ed7"
format_stamp: "Formatted at 2026-08-05 22:46:48 on dist-test-slave-7lbf"
server_key: "6b1e1f3fb03141491c9e8cfb73f48754"
server_key_iv: "81280abba29ac8d5b1b0dfb3fdfe9bcb"
server_key_version: "encryptionkey@0"
I20260805 22:46:48.994647 4974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260805 22:46:48.998558 4992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:49.000114 4974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260805 22:46:49.000305 4974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "6ab2e501c77b48a68451ed064e928ed7"
format_stamp: "Formatted at 2026-08-05 22:46:48 on dist-test-slave-7lbf"
server_key: "6b1e1f3fb03141491c9e8cfb73f48754"
server_key_iv: "81280abba29ac8d5b1b0dfb3fdfe9bcb"
server_key_version: "encryptionkey@0"
I20260805 22:46:49.000524 4974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:49.023461 4974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:49.028352 4974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:49.028658 4974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:49.047245 4974 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:41539
I20260805 22:46:49.047266 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:41539 every 8 connection(s)
I20260805 22:46:49.048702 4974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:46:49.052778 5064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:49.054661 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4974
I20260805 22:46:49.054908 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:46:49.055728 4208 external_mini_cluster.cc:1428] Setting key 413435159a1b6b6336b4a6d159dead7e
I20260805 22:46:49.065210 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7: Bootstrap starting.
I20260805 22:46:49.068652 5064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:49.069960 5064 log.cc:824] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7: Log is configured to *not* fsync() on all Append() calls
Aug 05 22:46:49 dist-test-slave-7lbf krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:49.073335 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7: No bootstrap required, opened a new log
I20260805 22:46:49.079865 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:49.058233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.4.28.62:41539)
0805 22:46:49.059091 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:49.059105 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:49.059150 (+ 45us) server_negotiation.cc:407] Connection header received
0805 22:46:49.060887 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:49.060912 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:49.061333 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:49.061749 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:49.063263 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:49.064621 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:49.065972 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:49.066683 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:49.066866 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:49.071354 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:49.071364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:49.071384 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:49.071424 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:49.074779 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:49.075837 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:49.075847 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:49.075856 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:49.076078 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:49.076544 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:49.076550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:49.076553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:49.077200 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:49.077443 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:49.078151 (+ 708us) server_negotiation.cc:299] Negotiation successful
0805 22:46:49.078637 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":347,"threads_started":1}
I20260805 22:46:49.082201 5064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab2e501c77b48a68451ed064e928ed7" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41539 } }
I20260805 22:46:49.082607 5064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:49.082712 5064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab2e501c77b48a68451ed064e928ed7, State: Initialized, Role: FOLLOWER
I20260805 22:46:49.083262 5064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab2e501c77b48a68451ed064e928ed7" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41539 } }
I20260805 22:46:49.083513 5064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:46:49.083637 5064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:46:49.083788 5064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:49.086081 5064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab2e501c77b48a68451ed064e928ed7" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41539 } }
I20260805 22:46:49.086663 5064 leader_election.cc:302] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ab2e501c77b48a68451ed064e928ed7; no voters:
I20260805 22:46:49.087267 5064 leader_election.cc:288] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:46:49.087373 5071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:46:49.088043 5071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [term 1 LEADER]: Becoming Leader. State: Replica: 6ab2e501c77b48a68451ed064e928ed7, State: Running, Role: LEADER
I20260805 22:46:49.088581 5071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab2e501c77b48a68451ed064e928ed7" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41539 } }
I20260805 22:46:49.089334 5064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:46:49.091503 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ab2e501c77b48a68451ed064e928ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab2e501c77b48a68451ed064e928ed7" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41539 } } }
I20260805 22:46:49.091903 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [sys.catalog]: This master's current role is: LEADER
I20260805 22:46:49.096490 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab2e501c77b48a68451ed064e928ed7. Latest consensus state: current_term: 1 leader_uuid: "6ab2e501c77b48a68451ed064e928ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab2e501c77b48a68451ed064e928ed7" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41539 } } }
I20260805 22:46:49.096972 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7 [sys.catalog]: This master's current role is: LEADER
I20260805 22:46:49.097066 5077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:46:49.102629 5077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:46:49.111572 5077 catalog_manager.cc:1347] Generated new cluster ID: fe6f009ac75f4605b8b4c4d21a1d8784
I20260805 22:46:49.111711 5077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:46:49.135017 5077 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:46:49.137097 5077 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:46:49.147049 5077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ab2e501c77b48a68451ed064e928ed7: Generated new TSK 0
I20260805 22:46:49.148257 5077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:46:49.201404 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41539
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:49.551846 5100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:49.552203 5100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:49.552340 5100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:49.559837 5100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:49.560001 5100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:49.560180 5100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:46:49.571152 5100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41539
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:49.572759 5100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:49.574553 5100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:49.589203 5110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:49.589171 5112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:49.589244 5109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:49.590488 5100 server_base.cc:1034] running on GCE node
I20260805 22:46:49.591192 5100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:49.592219 5100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:49.593557 5100 hybrid_clock.cc:648] HybridClock initialized: now 1785970009593458 us; error 64 us; skew 500 ppm
I20260805 22:46:49.593988 5100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:49 dist-test-slave-7lbf krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970009, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:49.597785 5100 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:46:49.599445 5100 webserver.cc:460] Webserver started at http://127.4.28.1:43767/ using document root <none> and password file <none>
I20260805 22:46:49.600207 5100 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:49.600338 5100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:49.600687 5100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:49.604028 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "e0691d81a1184de5aa5b15d0d9ff1856"
format_stamp: "Formatted at 2026-08-05 22:46:49 on dist-test-slave-7lbf"
server_key: "35b1f65a1e7fb3f72b26152d142f1fbb"
server_key_iv: "1d6293b5c8c11ee54b3ab1f6c2165e1d"
server_key_version: "encryptionkey@0"
I20260805 22:46:49.604951 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "e0691d81a1184de5aa5b15d0d9ff1856"
format_stamp: "Formatted at 2026-08-05 22:46:49 on dist-test-slave-7lbf"
server_key: "35b1f65a1e7fb3f72b26152d142f1fbb"
server_key_iv: "1d6293b5c8c11ee54b3ab1f6c2165e1d"
server_key_version: "encryptionkey@0"
I20260805 22:46:49.610821 5100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260805 22:46:49.615031 5122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:49.616811 5100 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:46:49.617043 5100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "e0691d81a1184de5aa5b15d0d9ff1856"
format_stamp: "Formatted at 2026-08-05 22:46:49 on dist-test-slave-7lbf"
server_key: "35b1f65a1e7fb3f72b26152d142f1fbb"
server_key_iv: "1d6293b5c8c11ee54b3ab1f6c2165e1d"
server_key_version: "encryptionkey@0"
I20260805 22:46:49.617288 5100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:49.636435 5100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:49.641412 5100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:49.641752 5100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:49.642858 5100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:46:49.644433 5100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:46:49.644526 5100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:49.644660 5100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:46:49.644716 5100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:49.677791 5100 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:45293
I20260805 22:46:49.677832 5268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:45293 every 8 connection(s)
I20260805 22:46:49.679185 5100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:46:49.685804 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 5100
I20260805 22:46:49.686028 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:46:49.686642 4208 external_mini_cluster.cc:1428] Setting key 1f9bdc70345599dd010c3f073e053591
Aug 05 22:46:49 dist-test-slave-7lbf krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970009, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:49.701766 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:49.682020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:34455 (local address 127.4.28.62:41539)
0805 22:46:49.682484 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:49.682490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:49.683536 (+ 1046us) server_negotiation.cc:407] Connection header received
0805 22:46:49.685549 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:49.685556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:49.685651 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:49.685758 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:49.687600 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:49.688383 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:49.690213 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:49.690732 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:49.690882 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:49.695087 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:49.695095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:49.695099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:49.695168 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:49.698047 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:49.699125 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:49.699130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:49.699133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:49.699256 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:49.699805 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:49.699808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:49.699810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:49.700083 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:49.700197 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:49.701241 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0805 22:46:49.701432 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260805 22:46:49.703493 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:49.682490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:41539 (local address 127.4.28.1:34455)
0805 22:46:49.683302 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:49.683372 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:49.685193 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:49.686087 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:49.686101 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:49.686723 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:49.687394 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:49.687414 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:49.688577 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:49.688581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:49.690047 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:49.690062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:49.691003 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:49.691013 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:49.691285 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:49.692002 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:49.692033 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:49.694785 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:49.698247 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:49.698260 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:49.698282 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:49.698994 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:49.699368 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:49.699373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:49.699377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:49.699677 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:49.700232 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:49.700243 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:49.700776 (+ 533us) client_negotiation.cc:769] Sending connection context
0805 22:46:49.701185 (+ 409us) client_negotiation.cc:240] Negotiation successful
0805 22:46:49.701857 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":313,"threads_started":1}
I20260805 22:46:49.705210 5269 heartbeater.cc:347] Connected to a master server at 127.4.28.62:41539
I20260805 22:46:49.705607 5269 heartbeater.cc:464] Registering TS with master...
I20260805 22:46:49.706481 5269 heartbeater.cc:511] Master 127.4.28.62:41539 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
I20260805 22:46:49.708725 5017 ts_manager.cc:194] Registered new tserver with Master: e0691d81a1184de5aa5b15d0d9ff1856 (127.4.28.1:45293)
I20260805 22:46:49.710353 5017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:34455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:46:49.737489 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41539
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:50.089672 5276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:50.090005 5276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:50.090143 5276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:50.097787 5276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:50.097944 5276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:50.098126 5276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:46:50.108451 5276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41539
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:50.109951 5276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:50.111522 5276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:50.125988 5282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:50.126070 5284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:50.126025 5287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:50.131475 5276 server_base.cc:1034] running on GCE node
I20260805 22:46:50.132269 5276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:50.133410 5276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:50.134697 5276 hybrid_clock.cc:648] HybridClock initialized: now 1785970010134619 us; error 54 us; skew 500 ppm
I20260805 22:46:50.135154 5276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:50 dist-test-slave-7lbf krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970010, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:50.138736 5276 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:46:50.140538 5276 webserver.cc:460] Webserver started at http://127.4.28.2:39075/ using document root <none> and password file <none>
I20260805 22:46:50.141368 5276 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:50.141546 5276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:50.141906 5276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:50.145196 5276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "d7c1020c4a724854ae316added4a59ce"
format_stamp: "Formatted at 2026-08-05 22:46:50 on dist-test-slave-7lbf"
server_key: "2a95dc32f2a6a655b3fae4d474232dd9"
server_key_iv: "1ba66c4834e1662458442925ee5aa46d"
server_key_version: "encryptionkey@0"
I20260805 22:46:50.146150 5276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "d7c1020c4a724854ae316added4a59ce"
format_stamp: "Formatted at 2026-08-05 22:46:50 on dist-test-slave-7lbf"
server_key: "2a95dc32f2a6a655b3fae4d474232dd9"
server_key_iv: "1ba66c4834e1662458442925ee5aa46d"
server_key_version: "encryptionkey@0"
I20260805 22:46:50.152180 5276 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260805 22:46:50.156330 5295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:50.158097 5276 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260805 22:46:50.158289 5276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "d7c1020c4a724854ae316added4a59ce"
format_stamp: "Formatted at 2026-08-05 22:46:50 on dist-test-slave-7lbf"
server_key: "2a95dc32f2a6a655b3fae4d474232dd9"
server_key_iv: "1ba66c4834e1662458442925ee5aa46d"
server_key_version: "encryptionkey@0"
I20260805 22:46:50.158502 5276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:50.183600 5276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:50.188252 5276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:50.188560 5276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:50.189702 5276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:46:50.191208 5276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:46:50.191300 5276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:50.191428 5276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:46:50.191483 5276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:50.224587 5276 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:42717
I20260805 22:46:50.224666 5445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:42717 every 8 connection(s)
I20260805 22:46:50.226120 5276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:46:50.231065 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 5276
I20260805 22:46:50.231297 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:46:50.231833 4208 external_mini_cluster.cc:1428] Setting key 00bff618d88c8c7f99d0cefe5e0907f3
Aug 05 22:46:50 dist-test-slave-7lbf krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970010, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:50.250020 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.228907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:39561 (local address 127.4.28.62:41539)
0805 22:46:50.229363 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:50.229368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:50.230385 (+ 1017us) server_negotiation.cc:407] Connection header received
0805 22:46:50.233021 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:50.233026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:50.233116 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:50.233260 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:50.235481 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.236281 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.238703 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.239173 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.239283 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.243710 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:50.243716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:50.243720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:50.243760 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:50.246441 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:50.247295 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:50.247300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:50.247302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:50.247455 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:50.247941 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:50.247945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:50.247947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:50.248242 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:50.248403 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:50.249288 (+ 885us) server_negotiation.cc:299] Negotiation successful
0805 22:46:50.249587 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":237,"threads_started":1}
I20260805 22:46:50.250958 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.229395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:41539 (local address 127.4.28.2:39561)
0805 22:46:50.230201 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:50.230252 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:50.232657 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:50.233588 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:50.233598 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:50.234312 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:50.235230 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.235261 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.236475 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.236482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.238530 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.238549 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.239437 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.239445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.239739 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.240536 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:50.240569 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:50.243399 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:50.246618 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:50.246629 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:50.246649 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:50.247141 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:50.247579 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:50.247585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:50.247589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:50.247805 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:50.248393 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:50.248402 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:50.248787 (+ 385us) client_negotiation.cc:769] Sending connection context
0805 22:46:50.249098 (+ 311us) client_negotiation.cc:240] Negotiation successful
0805 22:46:50.249621 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":286,"threads_started":1}
I20260805 22:46:50.252583 5448 heartbeater.cc:347] Connected to a master server at 127.4.28.62:41539
I20260805 22:46:50.252912 5448 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
I20260805 22:46:50.253736 5448 heartbeater.cc:511] Master 127.4.28.62:41539 requested a full tablet report, sending...
I20260805 22:46:50.255585 5017 ts_manager.cc:194] Registered new tserver with Master: d7c1020c4a724854ae316added4a59ce (127.4.28.2:42717)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
I20260805 22:46:50.256685 5017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:39561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:46:50.280889 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41539
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:50.624164 5459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:50.624534 5459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:50.624652 5459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:50.632221 5459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:50.632364 5459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:50.632503 5459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:46:50.642850 5459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41539
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:50.644340 5459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:50.645823 5459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:50.660100 5466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:50.660136 5469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:50.660167 5467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:50.660928 5459 server_base.cc:1034] running on GCE node
I20260805 22:46:50.661648 5459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:50.662772 5459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:50.663995 5459 hybrid_clock.cc:648] HybridClock initialized: now 1785970010663944 us; error 44 us; skew 500 ppm
I20260805 22:46:50.664373 5459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:50 dist-test-slave-7lbf krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970010, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:50.667913 5459 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:46:50.669700 5459 webserver.cc:460] Webserver started at http://127.4.28.3:37463/ using document root <none> and password file <none>
I20260805 22:46:50.670789 5459 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:50.670955 5459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:50.671370 5459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:50.674697 5459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "cc68c505cff14750a4843afd8b563cc0"
format_stamp: "Formatted at 2026-08-05 22:46:50 on dist-test-slave-7lbf"
server_key: "1f2bf0d2f4342335646a663ba9c08843"
server_key_iv: "77285fd4ad97c5e304d15afc56292391"
server_key_version: "encryptionkey@0"
I20260805 22:46:50.675524 5459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "cc68c505cff14750a4843afd8b563cc0"
format_stamp: "Formatted at 2026-08-05 22:46:50 on dist-test-slave-7lbf"
server_key: "1f2bf0d2f4342335646a663ba9c08843"
server_key_iv: "77285fd4ad97c5e304d15afc56292391"
server_key_version: "encryptionkey@0"
I20260805 22:46:50.682736 5459 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260805 22:46:50.687151 5483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:50.688740 5459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260805 22:46:50.688938 5459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "cc68c505cff14750a4843afd8b563cc0"
format_stamp: "Formatted at 2026-08-05 22:46:50 on dist-test-slave-7lbf"
server_key: "1f2bf0d2f4342335646a663ba9c08843"
server_key_iv: "77285fd4ad97c5e304d15afc56292391"
server_key_version: "encryptionkey@0"
I20260805 22:46:50.689193 5459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:50.705300 5459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:50.710525 5459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:50.710881 5459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:50.712086 5459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:46:50.713866 5459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:46:50.713861 5269 heartbeater.cc:503] Master 127.4.28.62:41539 was elected leader, sending a full tablet report...
I20260805 22:46:50.713986 5459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:50.714151 5459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:46:50.714233 5459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:50.748754 5459 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:37825
I20260805 22:46:50.748792 5640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:37825 every 8 connection(s)
I20260805 22:46:50.750248 5459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:46:50.753816 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 5459
I20260805 22:46:50.754042 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:46:50.754622 4208 external_mini_cluster.cc:1428] Setting key 3501daf8de1e091f4e404c1183eaa269
Aug 05 22:46:50 dist-test-slave-7lbf krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970010, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:50.772437 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.753048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:38597 (local address 127.4.28.62:41539)
0805 22:46:50.753493 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:50.753498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:50.754691 (+ 1193us) server_negotiation.cc:407] Connection header received
0805 22:46:50.756826 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:50.756833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:50.756926 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:50.757074 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:50.758820 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.759648 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.761432 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.761914 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.762041 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.766061 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:50.766067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:50.766070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:50.766112 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:50.768891 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:50.769890 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:50.769897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:50.769900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:50.770064 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:50.770598 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:50.770603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:50.770606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:50.770947 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:50.771086 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:50.771927 (+ 841us) server_negotiation.cc:299] Negotiation successful
0805 22:46:50.772153 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":247,"threads_started":1}
I20260805 22:46:50.773473 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.753627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:41539 (local address 127.4.28.3:38597)
0805 22:46:50.754504 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:50.754548 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:50.756515 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:50.757294 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:50.757304 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:50.758007 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:50.758641 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.758662 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.759832 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.759837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.761253 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.761269 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.762207 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.762217 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.762513 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.763267 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:50.763312 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:50.765772 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:50.769070 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:50.769083 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:50.769100 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:50.769740 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:50.770187 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:50.770191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:50.770193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:50.770448 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:50.771126 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:50.771133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:50.771534 (+ 401us) client_negotiation.cc:769] Sending connection context
0805 22:46:50.771790 (+ 256us) client_negotiation.cc:240] Negotiation successful
0805 22:46:50.772254 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":374,"threads_started":1}
I20260805 22:46:50.775038 5642 heartbeater.cc:347] Connected to a master server at 127.4.28.62:41539
I20260805 22:46:50.775357 5642 heartbeater.cc:464] Registering TS with master...
I20260805 22:46:50.776158 5642 heartbeater.cc:511] Master 127.4.28.62:41539 requested a full tablet report, sending...
I20260805 22:46:50.777837 5017 ts_manager.cc:194] Registered new tserver with Master: cc68c505cff14750a4843afd8b563cc0 (127.4.28.3:37825)
I20260805 22:46:50.778700 5017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:38597
I20260805 22:46:50.783480 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:46:50.809609 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.795095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.4.28.62:41539)
0805 22:46:50.795298 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:50.795304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:50.796219 (+ 915us) server_negotiation.cc:407] Connection header received
0805 22:46:50.796424 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:50.796430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:50.796520 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:50.796629 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:50.798370 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.799153 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.800552 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.801042 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.801166 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.803744 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:50.803749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:50.803753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:50.803795 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:50.806600 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:50.807399 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:50.807404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:50.807407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:50.807554 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:50.807972 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:50.807977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:50.807979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:50.808353 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:50.808603 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:50.808966 (+ 363us) server_negotiation.cc:299] Negotiation successful
0805 22:46:50.809217 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260805 22:46:50.819515 5017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:46:50.828249 5017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:46:50.869912 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.846516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:45293 (local address 127.0.0.1:41932)
0805 22:46:50.860097 (+ 13581us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:50.860150 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:50.860478 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:50.861316 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:50.861324 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:50.861354 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:50.861752 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.861766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.863941 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.863946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.866512 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.866534 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.868598 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.868604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.868934 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.869028 (+ 94us) client_negotiation.cc:769] Sending connection context
0805 22:46:50.869173 (+ 145us) client_negotiation.cc:240] Negotiation successful
0805 22:46:50.869394 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13336,"mutex_wait_us":41,"thread_start_us":13771,"threads_started":1}
I20260805 22:46:50.870810 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.846925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.4.28.1:45293)
0805 22:46:50.859656 (+ 12731us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:50.859666 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:50.860318 (+ 652us) server_negotiation.cc:407] Connection header received
0805 22:46:50.860736 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:50.860743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:50.861049 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:50.861274 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:50.861992 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.863732 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.866730 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.868340 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.868454 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.868709 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:50.870225 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0805 22:46:50.870429 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12538,"thread_start_us":12345,"threads_started":1}
I20260805 22:46:50.871758 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.847018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:37825 (local address 127.0.0.1:44162)
0805 22:46:50.860098 (+ 13080us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:50.860155 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:50.860832 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:50.862494 (+ 1662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:50.862500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:50.862530 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:50.862999 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.863012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.866704 (+ 3692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.866709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.868901 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.868917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.870935 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.870942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.871130 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.871165 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:46:50.871238 (+ 73us) client_negotiation.cc:240] Negotiation successful
0805 22:46:50.871384 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12926,"mutex_wait_us":35,"thread_start_us":12664,"threads_started":1}
I20260805 22:46:50.872920 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.861908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42717 (local address 127.0.0.1:55870)
0805 22:46:50.865765 (+ 3857us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:50.865786 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:50.865913 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:50.866513 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:50.866518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:50.866542 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:50.866997 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.867007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.868362 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.868368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.870599 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.870616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.872174 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.872181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.872348 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.872380 (+ 32us) client_negotiation.cc:769] Sending connection context
0805 22:46:50.872473 (+ 93us) client_negotiation.cc:240] Negotiation successful
0805 22:46:50.872605 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3712,"thread_start_us":3503,"threads_started":1}
I20260805 22:46:50.872932 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.847595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.4.28.3:37825)
0805 22:46:50.859255 (+ 11660us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:50.859267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:50.860397 (+ 1130us) server_negotiation.cc:407] Connection header received
0805 22:46:50.861080 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:50.861088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:50.861948 (+ 860us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:50.862172 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:50.863224 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.866535 (+ 3311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.869123 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.870694 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.872023 (+ 1329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.872274 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:50.872380 (+ 106us) server_negotiation.cc:299] Negotiation successful
0805 22:46:50.872577 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11406,"thread_start_us":11170,"threads_started":1}
I20260805 22:46:50.873585 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.861077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.4.28.2:42717)
0805 22:46:50.865328 (+ 4251us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:50.865338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:50.865900 (+ 562us) server_negotiation.cc:407] Connection header received
0805 22:46:50.866063 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:50.866069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:50.866293 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:50.866504 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:50.867183 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.868205 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.870828 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.871877 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.872015 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.872245 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:50.872809 (+ 564us) server_negotiation.cc:299] Negotiation successful
0805 22:46:50.873290 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4054,"thread_start_us":3918,"threads_started":1}
I20260805 22:46:50.876619 5551 tablet_service.cc:1467] Processing CreateTablet for tablet dac50f26df1d4ee98205806e4f278369 (DEFAULT_TABLE table=test-table [id=8aa17419ba7d408db07e8b92084e6e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:46:50.876072 5180 tablet_service.cc:1467] Processing CreateTablet for tablet dac50f26df1d4ee98205806e4f278369 (DEFAULT_TABLE table=test-table [id=8aa17419ba7d408db07e8b92084e6e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:46:50.877439 5358 tablet_service.cc:1467] Processing CreateTablet for tablet dac50f26df1d4ee98205806e4f278369 (DEFAULT_TABLE table=test-table [id=8aa17419ba7d408db07e8b92084e6e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:46:50.878401 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac50f26df1d4ee98205806e4f278369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:50.878872 5358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac50f26df1d4ee98205806e4f278369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:50.878890 5180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac50f26df1d4ee98205806e4f278369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:50.892563 5669 tablet_bootstrap.cc:492] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0: Bootstrap starting.
I20260805 22:46:50.895051 5671 tablet_bootstrap.cc:492] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce: Bootstrap starting.
I20260805 22:46:50.896617 5672 tablet_bootstrap.cc:492] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856: Bootstrap starting.
I20260805 22:46:50.897199 5669 tablet_bootstrap.cc:654] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:50.898762 5669 log.cc:824] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0: Log is configured to *not* fsync() on all Append() calls
I20260805 22:46:50.898847 5671 tablet_bootstrap.cc:654] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:50.899801 5672 tablet_bootstrap.cc:654] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:50.900115 5671 log.cc:824] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce: Log is configured to *not* fsync() on all Append() calls
I20260805 22:46:50.901113 5672 log.cc:824] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856: Log is configured to *not* fsync() on all Append() calls
I20260805 22:46:50.903499 5669 tablet_bootstrap.cc:492] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0: No bootstrap required, opened a new log
I20260805 22:46:50.903781 5669 ts_tablet_manager.cc:1397] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260805 22:46:50.904428 5672 tablet_bootstrap.cc:492] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856: No bootstrap required, opened a new log
I20260805 22:46:50.904542 5671 tablet_bootstrap.cc:492] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce: No bootstrap required, opened a new log
I20260805 22:46:50.904717 5672 ts_tablet_manager.cc:1397] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260805 22:46:50.904780 5671 ts_tablet_manager.cc:1397] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260805 22:46:50.914592 5669 raft_consensus.cc:348] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } }
I20260805 22:46:50.915064 5669 raft_consensus.cc:374] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:50.915184 5669 raft_consensus.cc:729] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc68c505cff14750a4843afd8b563cc0, State: Initialized, Role: FOLLOWER
I20260805 22:46:50.915589 5672 raft_consensus.cc:348] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } }
I20260805 22:46:50.915587 5671 raft_consensus.cc:348] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } }
I20260805 22:46:50.915851 5669 consensus_queue.cc:260] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } }
I20260805 22:46:50.916072 5671 raft_consensus.cc:374] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:50.916072 5672 raft_consensus.cc:374] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:50.916170 5671 raft_consensus.cc:729] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c1020c4a724854ae316added4a59ce, State: Initialized, Role: FOLLOWER
I20260805 22:46:50.916170 5672 raft_consensus.cc:729] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691d81a1184de5aa5b15d0d9ff1856, State: Initialized, Role: FOLLOWER
I20260805 22:46:50.916920 5671 consensus_queue.cc:260] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } }
I20260805 22:46:50.916976 5672 consensus_queue.cc:260] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } }
I20260805 22:46:50.917199 5642 heartbeater.cc:503] Master 127.4.28.62:41539 was elected leader, sending a full tablet report...
I20260805 22:46:50.918924 5672 ts_tablet_manager.cc:1428] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260805 22:46:50.919399 5448 heartbeater.cc:503] Master 127.4.28.62:41539 was elected leader, sending a full tablet report...
I20260805 22:46:50.919935 5669 ts_tablet_manager.cc:1428] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260805 22:46:50.921311 5671 ts_tablet_manager.cc:1428] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
W20260805 22:46:50.931770 5270 tablet.cc:2369] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:46:50.946203 5677 raft_consensus.cc:482] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:46:50.946539 5677 raft_consensus.cc:504] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } }
I20260805 22:46:50.948948 5677 leader_election.cc:288] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0691d81a1184de5aa5b15d0d9ff1856 (127.4.28.1:45293), d7c1020c4a724854ae316added4a59ce (127.4.28.2:42717)
I20260805 22:46:50.955610 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.949636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:46853 (local address 127.4.28.1:45293)
0805 22:46:50.949858 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:50.949865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:50.949885 (+ 20us) server_negotiation.cc:407] Connection header received
0805 22:46:50.949964 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:50.949969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:50.950065 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:50.950169 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:50.950627 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.951627 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.953603 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.954674 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.954782 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.954871 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:50.955252 (+ 381us) server_negotiation.cc:299] Negotiation successful
0805 22:46:50.955360 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260805 22:46:50.955693 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.949213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:45293 (local address 127.4.28.3:46853)
0805 22:46:50.949471 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:50.949507 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:50.949684 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:50.950222 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:50.950226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:50.950255 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:50.950448 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.950460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.951781 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.951787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.953400 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.953415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.954849 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.954855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.955063 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.955102 (+ 39us) client_negotiation.cc:769] Sending connection context
0805 22:46:50.955210 (+ 108us) client_negotiation.cc:240] Negotiation successful
0805 22:46:50.955353 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":44}
I20260805 22:46:50.956354 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.949552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:55255 (local address 127.4.28.2:42717)
0805 22:46:50.949785 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:50.949792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:50.950440 (+ 648us) server_negotiation.cc:407] Connection header received
0805 22:46:50.950571 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:50.950578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:50.950654 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:50.950726 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:50.951320 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.952266 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.954304 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.955361 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.955495 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.955596 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:50.955960 (+ 364us) server_negotiation.cc:299] Negotiation successful
0805 22:46:50.956109 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260805 22:46:50.956442 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.949393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42717 (local address 127.4.28.3:55255)
0805 22:46:50.950325 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:50.950340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:50.950448 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:50.950775 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:50.950780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:50.950806 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:50.951160 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.951173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.952440 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.952444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.954127 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:50.954143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.955611 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.955618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:50.955793 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.955825 (+ 32us) client_negotiation.cc:769] Sending connection context
0805 22:46:50.955920 (+ 95us) client_negotiation.cc:240] Negotiation successful
0805 22:46:50.956054 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":26,"thread_start_us":545,"threads_started":1}
I20260805 22:46:50.957087 5205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac50f26df1d4ee98205806e4f278369" candidate_uuid: "cc68c505cff14750a4843afd8b563cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" is_pre_election: true
I20260805 22:46:50.957563 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac50f26df1d4ee98205806e4f278369" candidate_uuid: "cc68c505cff14750a4843afd8b563cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c1020c4a724854ae316added4a59ce" is_pre_election: true
I20260805 22:46:50.958045 5205 raft_consensus.cc:2454] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc68c505cff14750a4843afd8b563cc0 in term 0.
I20260805 22:46:50.958127 5386 raft_consensus.cc:2454] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc68c505cff14750a4843afd8b563cc0 in term 0.
I20260805 22:46:50.960031 5484 leader_election.cc:302] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc68c505cff14750a4843afd8b563cc0, d7c1020c4a724854ae316added4a59ce; no voters:
I20260805 22:46:50.960492 5677 raft_consensus.cc:2781] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:46:50.960640 5677 raft_consensus.cc:482] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:46:50.960747 5677 raft_consensus.cc:3037] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:50.963347 5677 raft_consensus.cc:504] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } }
I20260805 22:46:50.964115 5677 leader_election.cc:288] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [CANDIDATE]: Term 1 election: Requested vote from peers e0691d81a1184de5aa5b15d0d9ff1856 (127.4.28.1:45293), d7c1020c4a724854ae316added4a59ce (127.4.28.2:42717)
I20260805 22:46:50.964556 5205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac50f26df1d4ee98205806e4f278369" candidate_uuid: "cc68c505cff14750a4843afd8b563cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691d81a1184de5aa5b15d0d9ff1856"
I20260805 22:46:50.964689 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac50f26df1d4ee98205806e4f278369" candidate_uuid: "cc68c505cff14750a4843afd8b563cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c1020c4a724854ae316added4a59ce"
I20260805 22:46:50.964805 5205 raft_consensus.cc:3037] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:50.964906 5386 raft_consensus.cc:3037] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:50.968221 5386 raft_consensus.cc:2454] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc68c505cff14750a4843afd8b563cc0 in term 1.
I20260805 22:46:50.968221 5205 raft_consensus.cc:2454] T dac50f26df1d4ee98205806e4f278369 P e0691d81a1184de5aa5b15d0d9ff1856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc68c505cff14750a4843afd8b563cc0 in term 1.
I20260805 22:46:50.968777 5487 leader_election.cc:302] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc68c505cff14750a4843afd8b563cc0, e0691d81a1184de5aa5b15d0d9ff1856; no voters:
I20260805 22:46:50.969161 5677 raft_consensus.cc:2781] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:46:50.969671 5677 raft_consensus.cc:686] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [term 1 LEADER]: Becoming Leader. State: Replica: cc68c505cff14750a4843afd8b563cc0, State: Running, Role: LEADER
I20260805 22:46:50.970224 5677 consensus_queue.cc:237] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } }
I20260805 22:46:50.976732 5016 catalog_manager.cc:5495] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc68c505cff14750a4843afd8b563cc0 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "cc68c505cff14750a4843afd8b563cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691d81a1184de5aa5b15d0d9ff1856" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc68c505cff14750a4843afd8b563cc0" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c1020c4a724854ae316added4a59ce" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42717 } health_report { overall_health: UNKNOWN } } }
W20260805 22:46:50.978196 5450 tablet.cc:2369] T dac50f26df1d4ee98205806e4f278369 P d7c1020c4a724854ae316added4a59ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 05 22:46:50 dist-test-slave-7lbf krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
W20260805 22:46:51.001969 5643 tablet.cc:2369] T dac50f26df1d4ee98205806e4f278369 P cc68c505cff14750a4843afd8b563cc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:46:51.004721 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:50.989262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.4.28.1:45293)
0805 22:46:50.989458 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:50.989463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:50.990070 (+ 607us) server_negotiation.cc:407] Connection header received
0805 22:46:50.990195 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:50.990199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:50.990288 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:50.990378 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:50.991940 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.992774 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.994209 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:50.994789 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:50.994892 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:50.999020 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:50.999038 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:50.999056 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:50.999100 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:51.001950 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.002833 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.002842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.002844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.003027 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.003391 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.003394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.003396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.003764 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:51.004057 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:51.004260 (+ 203us) server_negotiation.cc:299] Negotiation successful
0805 22:46:51.004443 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Aug 05 22:46:51 dist-test-slave-7lbf krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 05 22:46:51 dist-test-slave-7lbf krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:46:51.061746 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:51.045939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.4.28.1:45293)
0805 22:46:51.046223 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:51.046230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:51.046390 (+ 160us) server_negotiation.cc:407] Connection header received
0805 22:46:51.046562 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:51.046567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:51.046649 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:51.046763 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:51.048213 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.049345 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.050643 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.051299 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.051427 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:51.055481 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:51.055488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:51.055494 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:51.055555 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:51.059015 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.059718 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.059725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.059728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.059905 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.060285 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.060289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.060293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.060647 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:51.060907 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:51.061097 (+ 190us) server_negotiation.cc:299] Negotiation successful
0805 22:46:51.061336 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
Aug 05 22:46:51 dist-test-slave-7lbf krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 22:46:51 dist-test-slave-7lbf krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:46:51.116711 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:51.102174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.4.28.1:45293)
0805 22:46:51.102470 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:51.102478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:51.102651 (+ 173us) server_negotiation.cc:407] Connection header received
0805 22:46:51.102867 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:51.102872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:51.102945 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:51.103043 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:51.104462 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.105356 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.106688 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.107132 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.107231 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:51.111188 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:51.111194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:51.111197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:51.111253 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:51.114078 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.114861 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.114866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.114869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.115022 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.115370 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.115373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.115375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.115701 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:51.115892 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:51.116167 (+ 275us) server_negotiation.cc:299] Negotiation successful
0805 22:46:51.116363 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260805 22:46:51.119238 5157 tablet_service.cc:2908] Rejecting scan request for tablet dac50f26df1d4ee98205806e4f278369: Uninitialized: clean time has not yet been initialized
I20260805 22:46:51.123114 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 5100
I20260805 22:46:51.136741 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:51.124086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.4.28.1:45293)
0805 22:46:51.124314 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:51.124321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:51.124578 (+ 257us) server_negotiation.cc:407] Connection header received
0805 22:46:51.124718 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:51.124722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:51.124803 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:51.124885 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:51.126201 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.126963 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.128277 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.128716 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.128878 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:51.131176 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:51.131182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:51.131185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:51.131225 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:51.133928 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.134647 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.134653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.134656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.134934 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.135387 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.135391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.135393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.135713 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:51.135981 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:51.136242 (+ 261us) server_negotiation.cc:299] Negotiation successful
0805 22:46:51.136439 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:46:51.137672 5260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:46:51.227177 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 5100
I20260805 22:46:51.254007 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 5276
Aug 05 22:46:51 dist-test-slave-7lbf krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:46:51.270501 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:51.254888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.4.28.2:42717)
0805 22:46:51.255092 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:51.255097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:51.255118 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:46:51.255266 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:51.255270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:51.255395 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:51.255486 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:51.256712 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.257500 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.258937 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.259374 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.259487 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:51.264149 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:51.264172 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:51.264192 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:51.264243 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:51.267491 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.268315 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.268325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.268327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.268499 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.268834 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.268838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.268840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.269282 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:51.269661 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:51.269985 (+ 324us) server_negotiation.cc:299] Negotiation successful
0805 22:46:51.270184 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 22:46:51.271236 5438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:46:51.359865 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 5276
I20260805 22:46:51.383481 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 5459
Aug 05 22:46:51 dist-test-slave-7lbf krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:46:51.400466 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:51.384412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.4.28.3:37825)
0805 22:46:51.384896 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:51.384900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:51.384917 (+ 17us) server_negotiation.cc:407] Connection header received
0805 22:46:51.384975 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:51.384979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:51.385043 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:51.385139 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:51.386577 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.387376 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.388770 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:51.389211 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:51.389368 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:51.394128 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:51.394152 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:51.394172 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:51.394226 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:51.397150 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.398080 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.398090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.398093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.398256 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:51.398719 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:51.398724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:51.398726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:51.399154 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:51.399477 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:51.399795 (+ 318us) server_negotiation.cc:299] Negotiation successful
0805 22:46:51.400057 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":263,"threads_started":1}
I20260805 22:46:51.401259 5636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:46:51.491305 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 5459
I20260805 22:46:51.511598 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4974
I20260805 22:46:51.512866 5056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:46:51.622035 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4974
2026-08-05T22:46:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4447 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:46:51 dist-test-slave-7lbf krb5kdc[5709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:46:51 dist-test-slave-7lbf krb5kdc[5709](info): set up 2 sockets
Aug 05 22:46:51 dist-test-slave-7lbf krb5kdc[5709](info): commencing operation
krb5kdc: starting...
W20260805 22:46:52.884824 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260805 22:46:53.023607 4208 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.139s user 0.000s sys 0.008s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260805 22:46:53.162456 4208 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.139s user 0.002s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260805 22:46:53.298084 4208 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.135s user 0.000s sys 0.008s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/test-user.keytab.
W20260805 22:46:53.821062 4208 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.523s user 0.000s sys 0.009s
Aug 05 22:46:53 dist-test-slave-7lbf krb5kdc[5709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260805 22:46:54.011515 4208 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.190s user 0.002s sys 0.054s
2026-08-05T22:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:46:54.139073 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:36471
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:36471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:54.642158 5724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:54.642544 5724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:54.642684 5724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:54.650413 5724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:46:54.650578 5724 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260805 22:46:54.650642 5724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:54.650729 5724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:54.650794 5724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:46:54.650882 5724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:46:54.661484 5724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:36471
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:36471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:54.663009 5724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:54.664450 5724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:54.676918 5730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:54.676879 5733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:54.677157 5735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:54.678025 5724 server_base.cc:1034] running on GCE node
I20260805 22:46:54.678925 5724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:54.680518 5724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:54.681756 5724 hybrid_clock.cc:648] HybridClock initialized: now 1785970014681686 us; error 46 us; skew 500 ppm
I20260805 22:46:54.682123 5724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:54 dist-test-slave-7lbf krb5kdc[5709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970014, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:54.685586 5724 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:46:54.687243 5724 webserver.cc:460] Webserver started at http://127.4.28.62:38175/ using document root <none> and password file <none>
I20260805 22:46:54.687995 5724 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:54.688097 5724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:54.688396 5724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:54.691689 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "4ef4ebacfd83483d899e83868b9ec519"
format_stamp: "Formatted at 2026-08-05 22:46:54 on dist-test-slave-7lbf"
server_key: "95a71af35ccfbd07ff065c4bcd5e35fb"
server_key_iv: "913d92064e71402d9531bfef5a3bf8fa"
server_key_version: "encryptionkey@0"
I20260805 22:46:54.692564 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "4ef4ebacfd83483d899e83868b9ec519"
format_stamp: "Formatted at 2026-08-05 22:46:54 on dist-test-slave-7lbf"
server_key: "95a71af35ccfbd07ff065c4bcd5e35fb"
server_key_iv: "913d92064e71402d9531bfef5a3bf8fa"
server_key_version: "encryptionkey@0"
I20260805 22:46:54.698628 5724 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260805 22:46:54.702684 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:54.704281 5724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:46:54.704461 5724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "4ef4ebacfd83483d899e83868b9ec519"
format_stamp: "Formatted at 2026-08-05 22:46:54 on dist-test-slave-7lbf"
server_key: "95a71af35ccfbd07ff065c4bcd5e35fb"
server_key_iv: "913d92064e71402d9531bfef5a3bf8fa"
server_key_version: "encryptionkey@0"
I20260805 22:46:54.704674 5724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:54.746057 5724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:54.750512 5724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:54.750779 5724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:54.773346 5724 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:36471
I20260805 22:46:54.773386 5830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:36471 every 8 connection(s)
I20260805 22:46:54.774879 5724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:46:54.776811 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 5724
I20260805 22:46:54.777019 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:46:54.777753 4208 external_mini_cluster.cc:1428] Setting key bf8d30d976e5972dd52c7661e7741fd1
I20260805 22:46:54.779618 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:46:54 dist-test-slave-7lbf krb5kdc[5709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:54.796247 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519: Bootstrap starting.
I20260805 22:46:54.799732 5832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:54.800863 5832 log.cc:824] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519: Log is configured to *not* fsync() on all Append() calls
I20260805 22:46:54.804070 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519: No bootstrap required, opened a new log
I20260805 22:46:54.804718 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:54.779928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.4.28.62:36471)
0805 22:46:54.780809 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:54.780826 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:54.780892 (+ 66us) server_negotiation.cc:407] Connection header received
0805 22:46:54.783089 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:54.783123 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:54.783737 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:54.784276 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:54.785865 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:54.788034 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:54.790027 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:54.791230 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:54.791470 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:54.797009 (+ 5539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:54.797020 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:54.797040 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:54.797090 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:54.800230 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:54.801026 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:54.801037 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:54.801047 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:54.801247 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:54.801723 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:54.801727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:54.801733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:54.802345 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:54.802651 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:54.803149 (+ 498us) server_negotiation.cc:299] Negotiation successful
0805 22:46:54.803624 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":357,"threads_started":1}
I20260805 22:46:54.812399 5832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef4ebacfd83483d899e83868b9ec519" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 36471 } }
I20260805 22:46:54.812812 5832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:54.812911 5832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef4ebacfd83483d899e83868b9ec519, State: Initialized, Role: FOLLOWER
I20260805 22:46:54.813509 5832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef4ebacfd83483d899e83868b9ec519" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 36471 } }
I20260805 22:46:54.813824 5832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:46:54.813961 5832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:46:54.814133 5832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:54.816642 5832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef4ebacfd83483d899e83868b9ec519" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 36471 } }
I20260805 22:46:54.817248 5832 leader_election.cc:302] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ef4ebacfd83483d899e83868b9ec519; no voters:
I20260805 22:46:54.817855 5832 leader_election.cc:288] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:46:54.818001 5840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:46:54.818647 5840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [term 1 LEADER]: Becoming Leader. State: Replica: 4ef4ebacfd83483d899e83868b9ec519, State: Running, Role: LEADER
I20260805 22:46:54.819217 5840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef4ebacfd83483d899e83868b9ec519" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 36471 } }
I20260805 22:46:54.819979 5832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:46:54.823032 5841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ef4ebacfd83483d899e83868b9ec519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef4ebacfd83483d899e83868b9ec519" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 36471 } } }
I20260805 22:46:54.823483 5841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [sys.catalog]: This master's current role is: LEADER
I20260805 22:46:54.823318 5844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ef4ebacfd83483d899e83868b9ec519. Latest consensus state: current_term: 1 leader_uuid: "4ef4ebacfd83483d899e83868b9ec519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef4ebacfd83483d899e83868b9ec519" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 36471 } } }
I20260805 22:46:54.823580 5844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519 [sys.catalog]: This master's current role is: LEADER
I20260805 22:46:54.825605 5852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:46:54.830102 5852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:46:54.839419 5852 catalog_manager.cc:1347] Generated new cluster ID: 85123db07ef947a9adaaeb88385e2e56
I20260805 22:46:54.839550 5852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:46:54.851949 5852 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:46:54.853355 5852 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:46:54.862308 5852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519: Generated new TSK 0
I20260805 22:46:54.863523 5852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:46:54.926134 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:36471
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:55.287967 5871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:55.288276 5871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:55.288416 5871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:55.295953 5871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:55.296088 5871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:55.296213 5871 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 22:46:55.296319 5871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:46:55.307365 5871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.28.62:36471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:55.308911 5871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:55.310446 5871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:55.324071 5877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:55.324002 5882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:55.323995 5878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:55.324837 5871 server_base.cc:1034] running on GCE node
I20260805 22:46:55.325651 5871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:55.326624 5871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:55.327859 5871 hybrid_clock.cc:648] HybridClock initialized: now 1785970015327782 us; error 51 us; skew 500 ppm
I20260805 22:46:55.328188 5871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:55 dist-test-slave-7lbf krb5kdc[5709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970015, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:55.331703 5871 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:46:55.333416 5871 webserver.cc:460] Webserver started at http://127.4.28.1:34555/ using document root <none> and password file <none>
I20260805 22:46:55.334255 5871 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:55.334354 5871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:55.334663 5871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:55.337929 5871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "8c47b432cb734251bdded81a9470eda3"
format_stamp: "Formatted at 2026-08-05 22:46:55 on dist-test-slave-7lbf"
server_key: "0d6b81eee5b1183e7e9d615849c41961"
server_key_iv: "8d100a33d1468863ee780c4acf535fb0"
server_key_version: "encryptionkey@0"
I20260805 22:46:55.338822 5871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "8c47b432cb734251bdded81a9470eda3"
format_stamp: "Formatted at 2026-08-05 22:46:55 on dist-test-slave-7lbf"
server_key: "0d6b81eee5b1183e7e9d615849c41961"
server_key_iv: "8d100a33d1468863ee780c4acf535fb0"
server_key_version: "encryptionkey@0"
I20260805 22:46:55.344646 5871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:46:55.348598 5893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:55.350411 5871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:46:55.350625 5871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "8c47b432cb734251bdded81a9470eda3"
format_stamp: "Formatted at 2026-08-05 22:46:55 on dist-test-slave-7lbf"
server_key: "0d6b81eee5b1183e7e9d615849c41961"
server_key_iv: "8d100a33d1468863ee780c4acf535fb0"
server_key_version: "encryptionkey@0"
I20260805 22:46:55.350869 5871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:55.372427 5871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:55.377274 5871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:55.377620 5871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:55.380920 5871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:46:55.381043 5871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:55.381211 5871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:46:55.381289 5871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:46:55 dist-test-slave-7lbf krb5kdc[5709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970015, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:55.407418 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:55.381568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:58105 (local address 127.4.28.62:36471)
0805 22:46:55.382330 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:55.382335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:55.383699 (+ 1364us) server_negotiation.cc:407] Connection header received
0805 22:46:55.385899 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:55.385905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:55.386005 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:55.386106 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:55.388573 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.389451 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.392695 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.393251 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.393402 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:55.399082 (+ 5680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:55.399091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:55.399097 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:55.399152 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:55.402921 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:55.404363 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:55.404371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:55.404375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:55.404565 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:55.405348 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:55.405354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:55.405358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:55.405738 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:55.405887 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:55.406777 (+ 890us) server_negotiation.cc:299] Negotiation successful
0805 22:46:55.407036 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":529,"threads_started":1}
I20260805 22:46:55.408684 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:55.382439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:36471 (local address 127.4.28.1:58105)
0805 22:46:55.383485 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:55.383546 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:55.385508 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:55.386461 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:55.386480 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:55.387356 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:55.388345 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:55.388378 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.389690 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.389697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:55.392456 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:55.392485 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.393623 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.393638 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:55.394024 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:55.395085 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:55.395145 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:55.398698 (+ 3553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:55.403182 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:55.403195 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:55.403219 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:55.404120 (+ 901us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:55.404727 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:55.404732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:55.404740 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:55.405179 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:55.405888 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:55.405897 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:55.406361 (+ 464us) client_negotiation.cc:769] Sending connection context
0805 22:46:55.406657 (+ 296us) client_negotiation.cc:240] Negotiation successful
0805 22:46:55.407251 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":455,"threads_started":1}
I20260805 22:46:55.428112 5871 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:39219
I20260805 22:46:55.428184 6058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:39219 every 8 connection(s)
I20260805 22:46:55.429862 5871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:46:55.431648 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 5871
I20260805 22:46:55.431838 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:46:55.432442 4208 external_mini_cluster.cc:1428] Setting key 2741abc4cf9b321454b74b7263ee334b
I20260805 22:46:55.434069 6059 heartbeater.cc:347] Connected to a master server at 127.4.28.62:36471
I20260805 22:46:55.434353 6059 heartbeater.cc:464] Registering TS with master...
I20260805 22:46:55.435351 6059 heartbeater.cc:511] Master 127.4.28.62:36471 requested a full tablet report, sending...
I20260805 22:46:55.437414 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:55.418351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:44625 (local address 127.4.28.62:36471)
0805 22:46:55.418527 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:55.418532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:55.419174 (+ 642us) server_negotiation.cc:407] Connection header received
0805 22:46:55.419471 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:55.419478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:55.419582 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:55.419702 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:55.421699 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.422433 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.424875 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.425462 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.425635 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:55.428450 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:55.428458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:55.428462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:55.428507 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:55.433856 (+ 5349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:55.434871 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:55.434876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:55.434880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:55.435058 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:55.435561 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:55.435567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:55.435570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:55.436016 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:55.436205 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:55.436727 (+ 522us) server_negotiation.cc:299] Negotiation successful
0805 22:46:55.437034 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 22:46:55.437477 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:55.418361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:36471 (local address 127.4.28.1:44625)
0805 22:46:55.419046 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:55.419064 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:55.419233 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:55.419922 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:55.419928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:55.420772 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:55.421469 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:55.421491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.422610 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.422617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:55.424641 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:55.424668 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.425957 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.425968 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:55.426151 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:55.426881 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:55.426910 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:55.428184 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:55.434048 (+ 5864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:55.434058 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:55.434063 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:55.434676 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:55.435199 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:55.435209 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:55.435213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:55.435397 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:55.436182 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:55.436187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:55.436424 (+ 237us) client_negotiation.cc:769] Sending connection context
0805 22:46:55.436607 (+ 183us) client_negotiation.cc:240] Negotiation successful
0805 22:46:55.437017 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":390,"threads_started":1}
I20260805 22:46:55.438118 5770 ts_manager.cc:194] Registered new tserver with Master: 8c47b432cb734251bdded81a9470eda3 (127.4.28.1:39219)
I20260805 22:46:55.439697 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:58105
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:46:55.484776 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:36471
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:55.838791 6063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:55.839131 6063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:55.839236 6063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:55.846848 6063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:55.846995 6063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:55.847090 6063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 22:46:55.847174 6063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:46:55.857756 6063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.28.62:36471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:55.859162 6063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:55.860630 6063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:55.875633 6072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:55.875710 6075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:55.875742 6071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:55.876471 6063 server_base.cc:1034] running on GCE node
I20260805 22:46:55.877264 6063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:55.878412 6063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:55.879688 6063 hybrid_clock.cc:648] HybridClock initialized: now 1785970015879599 us; error 53 us; skew 500 ppm
I20260805 22:46:55.880096 6063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:55 dist-test-slave-7lbf krb5kdc[5709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970015, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:55.883697 6063 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:46:55.885383 6063 webserver.cc:460] Webserver started at http://127.4.28.2:44843/ using document root <none> and password file <none>
I20260805 22:46:55.886209 6063 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:55.886313 6063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:55.886629 6063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:55.889976 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "c4790cecfade452c833d9062784835a7"
format_stamp: "Formatted at 2026-08-05 22:46:55 on dist-test-slave-7lbf"
server_key: "a017da29a214f0f96a24579f814a7dff"
server_key_iv: "aa0f5765f492dcc461c9b2e6d22262fd"
server_key_version: "encryptionkey@0"
I20260805 22:46:55.890846 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "c4790cecfade452c833d9062784835a7"
format_stamp: "Formatted at 2026-08-05 22:46:55 on dist-test-slave-7lbf"
server_key: "a017da29a214f0f96a24579f814a7dff"
server_key_iv: "aa0f5765f492dcc461c9b2e6d22262fd"
server_key_version: "encryptionkey@0"
I20260805 22:46:55.896826 6063 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260805 22:46:55.900902 6086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:55.902488 6063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:46:55.902667 6063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "c4790cecfade452c833d9062784835a7"
format_stamp: "Formatted at 2026-08-05 22:46:55 on dist-test-slave-7lbf"
server_key: "a017da29a214f0f96a24579f814a7dff"
server_key_iv: "aa0f5765f492dcc461c9b2e6d22262fd"
server_key_version: "encryptionkey@0"
I20260805 22:46:55.902937 6063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:55.922101 6063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:55.927083 6063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:55.927413 6063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:55.930771 6063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:46:55.930904 6063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:55.931066 6063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:46:55.931144 6063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:46:55 dist-test-slave-7lbf krb5kdc[5709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970015, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:55.957449 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:55.931190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:60147 (local address 127.4.28.62:36471)
0805 22:46:55.931394 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:55.931398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:55.933362 (+ 1964us) server_negotiation.cc:407] Connection header received
0805 22:46:55.935820 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:55.935827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:55.935925 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:55.936022 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:55.939086 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.939843 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.942563 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.943498 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.943686 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:55.948866 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:55.948875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:55.948882 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:55.948939 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:55.952555 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:55.953813 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:55.953822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:55.953827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:55.954028 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:55.954867 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:55.954874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:55.954877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:55.955269 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:55.955441 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:55.956706 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0805 22:46:55.957014 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 22:46:55.958918 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:55.932093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:36471 (local address 127.4.28.2:60147)
0805 22:46:55.933153 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:55.933210 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:55.935363 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:55.936655 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:55.936669 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:55.937569 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:55.938850 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:55.938881 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.940051 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.940058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:55.942299 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:55.942318 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.943841 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.943852 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:55.944182 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:55.945057 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:55.945098 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:55.948469 (+ 3371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:55.952741 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:55.952758 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:55.952779 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:55.953602 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:55.954217 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:55.954222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:55.954229 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:55.954674 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:55.955534 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:55.955549 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:55.956178 (+ 629us) client_negotiation.cc:769] Sending connection context
0805 22:46:55.956553 (+ 375us) client_negotiation.cc:240] Negotiation successful
0805 22:46:55.957224 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":508,"threads_started":1}
I20260805 22:46:55.978775 6063 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:41441
I20260805 22:46:55.978794 6256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:41441 every 8 connection(s)
I20260805 22:46:55.980875 6063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:46:55.984170 6258 heartbeater.cc:347] Connected to a master server at 127.4.28.62:36471
I20260805 22:46:55.984450 6258 heartbeater.cc:464] Registering TS with master...
I20260805 22:46:55.985329 6258 heartbeater.cc:511] Master 127.4.28.62:36471 requested a full tablet report, sending...
I20260805 22:46:55.987036 5770 ts_manager.cc:194] Registered new tserver with Master: c4790cecfade452c833d9062784835a7 (127.4.28.2:41441)
I20260805 22:46:55.987864 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:60147
I20260805 22:46:55.989097 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 6063
I20260805 22:46:55.989259 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:46:55.989842 4208 external_mini_cluster.cc:1428] Setting key 8a3df003883edad3400e7db5ab6057d5
I20260805 22:46:55.993837 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:55.979378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:36471 (local address 127.4.28.2:57187)
0805 22:46:55.980153 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:55.980174 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:55.980498 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:55.980876 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:55.980881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:55.981499 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:55.982108 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:55.982127 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.983613 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.983617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:55.984764 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:55.984779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.985863 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.985870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:55.986020 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:55.986704 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:55.986733 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:55.987502 (+ 769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:55.990778 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:55.990784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:55.990788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:55.991369 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:55.991960 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:55.991965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:55.991969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:55.992104 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:55.992885 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:55.992890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:55.993055 (+ 165us) client_negotiation.cc:769] Sending connection context
0805 22:46:55.993175 (+ 120us) client_negotiation.cc:240] Negotiation successful
0805 22:46:55.993447 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":392,"threads_started":1}
I20260805 22:46:55.993907 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:55.979417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:57187 (local address 127.4.28.62:36471)
0805 22:46:55.979648 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:55.979656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:55.980312 (+ 656us) server_negotiation.cc:407] Connection header received
0805 22:46:55.980650 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:55.980654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:55.980724 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:55.980810 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:55.982353 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.983198 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.984925 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:55.985622 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:55.985774 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:55.987698 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:55.987704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:55.987708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:55.987742 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:55.990613 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:55.991506 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:55.991512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:55.991516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:55.991807 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:55.992233 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:55.992239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:55.992243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:55.992725 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:55.992893 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:55.993295 (+ 402us) server_negotiation.cc:299] Negotiation successful
0805 22:46:55.993572 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:46:56.041141 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:36471
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:56.387112 6263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:56.387436 6263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:56.387564 6263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:56.395236 6263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:56.395375 6263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:56.395485 6263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 22:46:56.395596 6263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:46:56.406452 6263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.28.62:36471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:56.408021 6263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:56.409695 6263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:56.424052 6270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:56.424080 6274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:56.424036 6271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:56.425433 6263 server_base.cc:1034] running on GCE node
I20260805 22:46:56.426241 6263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:56.427387 6263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:56.428695 6263 hybrid_clock.cc:648] HybridClock initialized: now 1785970016428596 us; error 61 us; skew 500 ppm
I20260805 22:46:56.429071 6263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:56 dist-test-slave-7lbf krb5kdc[5709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970016, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:56.433452 6263 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:46:56.435163 6263 webserver.cc:460] Webserver started at http://127.4.28.3:35655/ using document root <none> and password file <none>
I20260805 22:46:56.435921 6263 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:56.436024 6263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:56.436333 6263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:56.439471 6263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "b8738a9446a149e1874dd162e326dee2"
format_stamp: "Formatted at 2026-08-05 22:46:56 on dist-test-slave-7lbf"
server_key: "31084704631e80d34fcbca106b68bf1c"
server_key_iv: "207c770044b3d9bdbe14aecb480d6e14"
server_key_version: "encryptionkey@0"
I20260805 22:46:56.440305 6263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "b8738a9446a149e1874dd162e326dee2"
format_stamp: "Formatted at 2026-08-05 22:46:56 on dist-test-slave-7lbf"
server_key: "31084704631e80d34fcbca106b68bf1c"
server_key_iv: "207c770044b3d9bdbe14aecb480d6e14"
server_key_version: "encryptionkey@0"
I20260805 22:46:56.444352 6059 heartbeater.cc:503] Master 127.4.28.62:36471 was elected leader, sending a full tablet report...
I20260805 22:46:56.446319 6263 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260805 22:46:56.450531 6284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:56.452303 6263 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:46:56.452476 6263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "b8738a9446a149e1874dd162e326dee2"
format_stamp: "Formatted at 2026-08-05 22:46:56 on dist-test-slave-7lbf"
server_key: "31084704631e80d34fcbca106b68bf1c"
server_key_iv: "207c770044b3d9bdbe14aecb480d6e14"
server_key_version: "encryptionkey@0"
I20260805 22:46:56.452684 6263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:56.476341 6263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:56.480679 6263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:56.480933 6263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:56.483844 6263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:46:56.484025 6263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260805 22:46:56.484179 6263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:46:56.484253 6263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:46:56 dist-test-slave-7lbf krb5kdc[5709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970016, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:56.512410 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.485389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:36471 (local address 127.4.28.3:54073)
0805 22:46:56.486385 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:56.486438 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:56.488496 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:56.489372 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:56.489382 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:56.490209 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:56.491142 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.491174 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.493004 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.493010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.495379 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.495401 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.496570 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.496577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.496794 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.497666 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:56.497698 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:56.501494 (+ 3796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:56.505867 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:56.505881 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:56.505908 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:56.506840 (+ 932us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:56.507473 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:56.507484 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:56.507488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:56.507844 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:56.508831 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:56.508843 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:56.509476 (+ 633us) client_negotiation.cc:769] Sending connection context
0805 22:46:56.509962 (+ 486us) client_negotiation.cc:240] Negotiation successful
0805 22:46:56.510732 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":380,"threads_started":1}
I20260805 22:46:56.513613 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.484661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:54073 (local address 127.4.28.62:36471)
0805 22:46:56.484869 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:56.484875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:56.486578 (+ 1703us) server_negotiation.cc:407] Connection header received
0805 22:46:56.488846 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:56.488853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:56.488957 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:56.489056 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:56.491499 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.492778 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.495595 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.496275 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.496460 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.501921 (+ 5461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:56.501930 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:56.501935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:56.501987 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:56.505619 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:56.507055 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:56.507064 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:56.507068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:56.507326 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:56.508054 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:56.508060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:56.508064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:56.508550 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:56.508719 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:56.513069 (+ 4350us) server_negotiation.cc:299] Negotiation successful
0805 22:46:56.513299 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:46:56.533434 6263 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:46577
I20260805 22:46:56.533464 6460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:46577 every 8 connection(s)
I20260805 22:46:56.534879 6263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:46:56.535243 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 6263
I20260805 22:46:56.535444 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:46:56.536010 4208 external_mini_cluster.cc:1428] Setting key 1b226d2e4934aaf965e1e03a41429536
I20260805 22:46:56.536216 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.520064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:36471 (local address 127.4.28.3:57003)
0805 22:46:56.521378 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:56.521401 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:56.521652 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:56.521995 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:56.522000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:56.522673 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:56.523133 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.523149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.524519 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.524525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.526484 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.526507 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.527709 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.527718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.527965 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.528679 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:56.528707 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:56.529921 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:56.533243 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:56.533249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:56.533254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:56.533824 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:56.534215 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:56.534220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:56.534224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:56.534358 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:56.534981 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:56.534986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:56.535205 (+ 219us) client_negotiation.cc:769] Sending connection context
0805 22:46:56.535342 (+ 137us) client_negotiation.cc:240] Negotiation successful
0805 22:46:56.535720 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"thread_start_us":912,"threads_started":1}
I20260805 22:46:56.537328 6461 heartbeater.cc:347] Connected to a master server at 127.4.28.62:36471
I20260805 22:46:56.537431 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.519977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:57003 (local address 127.4.28.62:36471)
0805 22:46:56.520196 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:56.520202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:56.521558 (+ 1356us) server_negotiation.cc:407] Connection header received
0805 22:46:56.521781 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:56.521786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:56.521857 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:56.521947 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:56.523419 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.524334 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.526708 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.527276 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.527448 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.530145 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:56.530151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:56.530154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:56.530196 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:56.533095 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:56.533955 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:56.533961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:56.533965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:56.534104 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:56.534484 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:56.534489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:56.534491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:56.534815 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:56.536626 (+ 1811us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:56.536822 (+ 196us) server_negotiation.cc:299] Negotiation successful
0805 22:46:56.537092 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:46:56.538273 6461 heartbeater.cc:464] Registering TS with master...
I20260805 22:46:56.540246 6461 heartbeater.cc:511] Master 127.4.28.62:36471 requested a full tablet report, sending...
I20260805 22:46:56.542517 5770 ts_manager.cc:194] Registered new tserver with Master: b8738a9446a149e1874dd162e326dee2 (127.4.28.3:46577)
I20260805 22:46:56.543354 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:54073
I20260805 22:46:56.553166 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 22:46:56 dist-test-slave-7lbf krb5kdc[5709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970016, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 05 22:46:56 dist-test-slave-7lbf krb5kdc[5709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970016, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:56.608238 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.593562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.4.28.62:36471)
0805 22:46:56.593736 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:56.593741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:56.594077 (+ 336us) server_negotiation.cc:407] Connection header received
0805 22:46:56.594242 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:56.594246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:56.594329 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:56.594427 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:56.595865 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.596620 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.598029 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.598454 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.598561 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.602687 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:56.602693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:56.602697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:56.602740 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:56.605381 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:56.606114 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:56.606119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:56.606121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:56.606263 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:56.606751 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:56.606756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:56.606759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:56.607085 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:56.607281 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:56.607648 (+ 367us) server_negotiation.cc:299] Negotiation successful
0805 22:46:56.607893 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 22:46:56.612603 5770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:46:56.620110 5770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:46:56.644500 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.632992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:46577 (local address 127.0.0.1:43092)
0805 22:46:56.634037 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:56.634097 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:56.634316 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:56.636161 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:56.636171 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:56.636209 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:56.636630 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.636643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.638735 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.638741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.641139 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.641167 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.643400 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.643408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.643652 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.643725 (+ 73us) client_negotiation.cc:769] Sending connection context
0805 22:46:56.643868 (+ 143us) client_negotiation.cc:240] Negotiation successful
0805 22:46:56.644104 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"threads_started":1}
I20260805 22:46:56.646481 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.633192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.4.28.3:46577)
0805 22:46:56.635079 (+ 1887us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:56.635089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:56.635126 (+ 37us) server_negotiation.cc:407] Connection header received
0805 22:46:56.635245 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:56.635255 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:56.635553 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:56.635765 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:56.636868 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.638559 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.641442 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.643115 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.645290 (+ 2175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.645626 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:56.645725 (+ 99us) server_negotiation.cc:299] Negotiation successful
0805 22:46:56.645934 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1660,"thread_start_us":1720,"threads_started":1}
I20260805 22:46:56.657356 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.633106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:41441 (local address 127.0.0.1:32928)
0805 22:46:56.644929 (+ 11823us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:56.644953 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:56.645107 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:56.647535 (+ 2428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:56.647541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:56.647574 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:56.647766 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.647776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.651845 (+ 4069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.651856 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.653855 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.653874 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.656225 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.656231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.656543 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.656584 (+ 41us) client_negotiation.cc:769] Sending connection context
0805 22:46:56.656692 (+ 108us) client_negotiation.cc:240] Negotiation successful
0805 22:46:56.656933 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11653,"mutex_wait_us":99,"thread_start_us":908,"threads_started":1}
I20260805 22:46:56.657485 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.633785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.4.28.2:41441)
0805 22:46:56.645627 (+ 11842us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:56.645638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:56.645677 (+ 39us) server_negotiation.cc:407] Connection header received
0805 22:46:56.645812 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:56.645822 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:56.646136 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:56.648007 (+ 1871us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:56.648203 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.649434 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.654496 (+ 5062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.656003 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.656159 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.656869 (+ 710us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:56.656977 (+ 108us) server_negotiation.cc:299] Negotiation successful
0805 22:46:56.657164 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11661,"thread_start_us":11528,"threads_started":1}
I20260805 22:46:56.657963 6356 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a1ba38afd48d6a802e14092434e0e (DEFAULT_TABLE table=test-table [id=e5d10c958a1342f3811aa854b80d4f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:46:56.659866 6356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a1ba38afd48d6a802e14092434e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:56.660882 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.635126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:39219 (local address 127.0.0.1:33782)
0805 22:46:56.652113 (+ 16987us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:56.652138 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:56.652308 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:56.653623 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:56.653628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:56.653653 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:56.654150 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.654166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.656224 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.656231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.658329 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.658347 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.660018 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.660025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.660202 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.660238 (+ 36us) client_negotiation.cc:769] Sending connection context
0805 22:46:56.660340 (+ 102us) client_negotiation.cc:240] Negotiation successful
0805 22:46:56.660504 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16832,"mutex_wait_us":22,"thread_start_us":17173,"threads_started":1}
I20260805 22:46:56.660902 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.635533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.4.28.1:39219)
0805 22:46:56.651467 (+ 15934us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:56.651478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:56.652844 (+ 1366us) server_negotiation.cc:407] Connection header received
0805 22:46:56.653010 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:56.653020 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:56.653351 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:56.654417 (+ 1066us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:56.654670 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.656065 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.658538 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.659774 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.659924 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.660145 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:56.660413 (+ 268us) server_negotiation.cc:299] Negotiation successful
0805 22:46:56.660609 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15684,"thread_start_us":15942,"threads_started":1}
I20260805 22:46:56.662686 6149 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a1ba38afd48d6a802e14092434e0e (DEFAULT_TABLE table=test-table [id=e5d10c958a1342f3811aa854b80d4f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:46:56.664026 6149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a1ba38afd48d6a802e14092434e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:56.665068 5965 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a1ba38afd48d6a802e14092434e0e (DEFAULT_TABLE table=test-table [id=e5d10c958a1342f3811aa854b80d4f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:46:56.666643 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6a1ba38afd48d6a802e14092434e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:56.676501 6491 tablet_bootstrap.cc:492] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: Bootstrap starting.
I20260805 22:46:56.678597 6492 tablet_bootstrap.cc:492] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: Bootstrap starting.
I20260805 22:46:56.680522 6491 tablet_bootstrap.cc:654] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:56.682106 6491 log.cc:824] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: Log is configured to *not* fsync() on all Append() calls
I20260805 22:46:56.682579 6492 tablet_bootstrap.cc:654] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:56.682914 6493 tablet_bootstrap.cc:492] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: Bootstrap starting.
I20260805 22:46:56.684029 6492 log.cc:824] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: Log is configured to *not* fsync() on all Append() calls
I20260805 22:46:56.686784 6493 tablet_bootstrap.cc:654] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:56.687129 6491 tablet_bootstrap.cc:492] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: No bootstrap required, opened a new log
I20260805 22:46:56.687374 6492 tablet_bootstrap.cc:492] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: No bootstrap required, opened a new log
I20260805 22:46:56.687404 6491 ts_tablet_manager.cc:1397] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260805 22:46:56.687682 6492 ts_tablet_manager.cc:1397] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260805 22:46:56.688264 6493 log.cc:824] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: Log is configured to *not* fsync() on all Append() calls
I20260805 22:46:56.695933 6491 raft_consensus.cc:348] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } }
I20260805 22:46:56.696349 6491 raft_consensus.cc:374] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:56.696481 6491 raft_consensus.cc:729] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8738a9446a149e1874dd162e326dee2, State: Initialized, Role: FOLLOWER
I20260805 22:46:56.697085 6493 tablet_bootstrap.cc:492] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: No bootstrap required, opened a new log
I20260805 22:46:56.697119 6491 consensus_queue.cc:260] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } }
I20260805 22:46:56.697420 6493 ts_tablet_manager.cc:1397] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260805 22:46:56.698803 6492 raft_consensus.cc:348] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } }
I20260805 22:46:56.699415 6492 raft_consensus.cc:374] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:56.699541 6492 raft_consensus.cc:729] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4790cecfade452c833d9062784835a7, State: Initialized, Role: FOLLOWER
I20260805 22:46:56.700234 6492 consensus_queue.cc:260] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } }
I20260805 22:46:56.701642 6258 heartbeater.cc:503] Master 127.4.28.62:36471 was elected leader, sending a full tablet report...
I20260805 22:46:56.704062 6492 ts_tablet_manager.cc:1428] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260805 22:46:56.705228 6461 heartbeater.cc:503] Master 127.4.28.62:36471 was elected leader, sending a full tablet report...
I20260805 22:46:56.706113 6491 ts_tablet_manager.cc:1428] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
I20260805 22:46:56.709291 6493 raft_consensus.cc:348] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } }
I20260805 22:46:56.709884 6493 raft_consensus.cc:374] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:56.710012 6493 raft_consensus.cc:729] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c47b432cb734251bdded81a9470eda3, State: Initialized, Role: FOLLOWER
I20260805 22:46:56.710755 6493 consensus_queue.cc:260] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } }
I20260805 22:46:56.712486 6493 ts_tablet_manager.cc:1428] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: Time spent starting tablet: real 0.015s user 0.005s sys 0.008s
W20260805 22:46:56.733551 6259 tablet.cc:2369] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:46:56.788810 6503 raft_consensus.cc:482] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:46:56.789180 6503 raft_consensus.cc:504] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } }
I20260805 22:46:56.791628 6503 leader_election.cc:288] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8738a9446a149e1874dd162e326dee2 (127.4.28.3:46577), 8c47b432cb734251bdded81a9470eda3 (127.4.28.1:39219)
W20260805 22:46:56.793397 6463 tablet.cc:2369] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:46:56.799468 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.792831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:58955 (local address 127.4.28.3:46577)
0805 22:46:56.793090 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:56.793109 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:56.793925 (+ 816us) server_negotiation.cc:407] Connection header received
0805 22:46:56.794228 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:56.794234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:56.794325 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:56.794408 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:56.795014 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.795738 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.797663 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.798486 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.798610 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.798727 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:56.799081 (+ 354us) server_negotiation.cc:299] Negotiation successful
0805 22:46:56.799210 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 22:46:56.799541 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.792526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:38713 (local address 127.4.28.1:39219)
0805 22:46:56.793028 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:56.793037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:56.794230 (+ 1193us) server_negotiation.cc:407] Connection header received
0805 22:46:56.794516 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:56.794523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:56.794604 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:56.794681 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:56.795254 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.795982 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.797823 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.798599 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.798740 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.798856 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:56.799147 (+ 291us) server_negotiation.cc:299] Negotiation successful
0805 22:46:56.799285 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260805 22:46:56.799645 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.791804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:46577 (local address 127.4.28.2:58955)
0805 22:46:56.793780 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:56.793806 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:56.794055 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:56.794456 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:56.794461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:56.794490 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:56.794855 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.794868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.795882 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.795888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.797472 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.797487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.798732 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.798739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.798917 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.798942 (+ 25us) client_negotiation.cc:769] Sending connection context
0805 22:46:56.799028 (+ 86us) client_negotiation.cc:240] Negotiation successful
0805 22:46:56.799198 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1809,"mutex_wait_us":1,"thread_start_us":1915,"threads_started":1}
I20260805 22:46:56.799697 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.792599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:39219 (local address 127.4.28.2:38713)
0805 22:46:56.794101 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:56.794128 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:56.794367 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:56.794727 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:56.794732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:56.794757 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:56.795097 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.795106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.796128 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.796134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.797659 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:56.797674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.798840 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.798846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:56.798998 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.799022 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 22:46:56.799104 (+ 82us) client_negotiation.cc:240] Negotiation successful
0805 22:46:56.799279 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1345,"mutex_wait_us":28,"thread_start_us":661,"threads_started":1}
I20260805 22:46:56.800827 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a1ba38afd48d6a802e14092434e0e" candidate_uuid: "c4790cecfade452c833d9062784835a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c47b432cb734251bdded81a9470eda3" is_pre_election: true
I20260805 22:46:56.800815 6387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a1ba38afd48d6a802e14092434e0e" candidate_uuid: "c4790cecfade452c833d9062784835a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8738a9446a149e1874dd162e326dee2" is_pre_election: true
I20260805 22:46:56.801338 6387 raft_consensus.cc:2454] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4790cecfade452c833d9062784835a7 in term 0.
I20260805 22:46:56.801339 5987 raft_consensus.cc:2454] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4790cecfade452c833d9062784835a7 in term 0.
I20260805 22:46:56.802042 6088 leader_election.cc:302] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8738a9446a149e1874dd162e326dee2, c4790cecfade452c833d9062784835a7; no voters:
I20260805 22:46:56.802459 6503 raft_consensus.cc:2781] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:46:56.802623 6503 raft_consensus.cc:482] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:46:56.802736 6503 raft_consensus.cc:3037] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:56.804971 6503 raft_consensus.cc:504] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } }
I20260805 22:46:56.805702 6503 leader_election.cc:288] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [CANDIDATE]: Term 1 election: Requested vote from peers b8738a9446a149e1874dd162e326dee2 (127.4.28.3:46577), 8c47b432cb734251bdded81a9470eda3 (127.4.28.1:39219)
I20260805 22:46:56.806205 6387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a1ba38afd48d6a802e14092434e0e" candidate_uuid: "c4790cecfade452c833d9062784835a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8738a9446a149e1874dd162e326dee2"
I20260805 22:46:56.806288 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a1ba38afd48d6a802e14092434e0e" candidate_uuid: "c4790cecfade452c833d9062784835a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c47b432cb734251bdded81a9470eda3"
I20260805 22:46:56.806411 6387 raft_consensus.cc:3037] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:56.806514 5987 raft_consensus.cc:3037] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:56.809000 6387 raft_consensus.cc:2454] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4790cecfade452c833d9062784835a7 in term 1.
I20260805 22:46:56.809077 5987 raft_consensus.cc:2454] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4790cecfade452c833d9062784835a7 in term 1.
I20260805 22:46:56.809502 6088 leader_election.cc:302] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8738a9446a149e1874dd162e326dee2, c4790cecfade452c833d9062784835a7; no voters:
I20260805 22:46:56.809922 6503 raft_consensus.cc:2781] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:46:56.810408 6503 raft_consensus.cc:686] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 1 LEADER]: Becoming Leader. State: Replica: c4790cecfade452c833d9062784835a7, State: Running, Role: LEADER
I20260805 22:46:56.810905 6503 consensus_queue.cc:237] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } }
I20260805 22:46:56.815914 5769 catalog_manager.cc:5495] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4790cecfade452c833d9062784835a7 (127.4.28.2). New cstate: current_term: 1 leader_uuid: "c4790cecfade452c833d9062784835a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } health_report { overall_health: UNKNOWN } } }
I20260805 22:46:56.880007 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:56.874010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.4.28.2:41441)
0805 22:46:56.874229 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:56.874235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:56.874367 (+ 132us) server_negotiation.cc:407] Connection header received
0805 22:46:56.874784 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:56.874789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:56.874874 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:56.875016 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:46:56.875388 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.876163 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.877552 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:56.877968 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:56.878110 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:56.878676 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:46:56.879331 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:46:56.879485 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:56.879627 (+ 142us) server_negotiation.cc:299] Negotiation successful
0805 22:46:56.879723 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260805 22:46:56.890344 5987 raft_consensus.cc:1261] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [term 1 FOLLOWER]: Refusing update from remote peer c4790cecfade452c833d9062784835a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:46:56.891697 6510 consensus_queue.cc:1035] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:46:56.892714 6387 raft_consensus.cc:1261] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Refusing update from remote peer c4790cecfade452c833d9062784835a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:46:56.893659 6510 consensus_queue.cc:1035] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:46:56.934659 5769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:46:56.935075 5769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:46:56.939191 5769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519: Sending DeleteTablet for 3 replicas of tablet 8e6a1ba38afd48d6a802e14092434e0e
I20260805 22:46:56.941301 5965 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6a1ba38afd48d6a802e14092434e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:46:56 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:33782
I20260805 22:46:56.941670 6356 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6a1ba38afd48d6a802e14092434e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:46:56 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:43092
I20260805 22:46:56.942395 6149 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6a1ba38afd48d6a802e14092434e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:46:56 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:32928
I20260805 22:46:56.947415 5770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:46:56.948292 5770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:46:56.958717 6529 tablet_replica.cc:330] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: stopping tablet replica
I20260805 22:46:56.958770 6355 tablet_service.cc:1467] Processing CreateTablet for tablet 7385cac5b23b4be5a80beee454424d63 (DEFAULT_TABLE table=test-table [id=106909d703504f50be5bf535760afb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:46:56.959352 6355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7385cac5b23b4be5a80beee454424d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:56.959971 6529 raft_consensus.cc:2229] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:46:56.960657 6529 raft_consensus.cc:2258] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:46:56.961711 6531 tablet_replica.cc:330] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: stopping tablet replica
I20260805 22:46:56.963346 5965 tablet_service.cc:1467] Processing CreateTablet for tablet 7385cac5b23b4be5a80beee454424d63 (DEFAULT_TABLE table=test-table [id=106909d703504f50be5bf535760afb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:46:56.963828 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7385cac5b23b4be5a80beee454424d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:56.964881 6529 ts_tablet_manager.cc:1905] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:46:56.966558 6491 tablet_bootstrap.cc:492] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: Bootstrap starting.
I20260805 22:46:56.967949 6491 tablet_bootstrap.cc:654] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:56.969971 6149 tablet_service.cc:1467] Processing CreateTablet for tablet 7385cac5b23b4be5a80beee454424d63 (DEFAULT_TABLE table=test-table [id=106909d703504f50be5bf535760afb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:46:56.970101 6491 tablet_bootstrap.cc:492] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: No bootstrap required, opened a new log
I20260805 22:46:56.970268 6491 ts_tablet_manager.cc:1397] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260805 22:46:56.970615 6149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7385cac5b23b4be5a80beee454424d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:56.971174 6491 raft_consensus.cc:348] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:56.971433 6491 raft_consensus.cc:374] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:56.971546 6491 raft_consensus.cc:729] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8738a9446a149e1874dd162e326dee2, State: Initialized, Role: FOLLOWER
I20260805 22:46:56.971860 6491 consensus_queue.cc:260] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:56.972477 6491 ts_tablet_manager.cc:1428] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260805 22:46:56.975817 6529 ts_tablet_manager.cc:1918] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:46:56.976110 6529 log.cc:1196] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/8e6a1ba38afd48d6a802e14092434e0e
I20260805 22:46:56.976794 6529 ts_tablet_manager.cc:1939] T 8e6a1ba38afd48d6a802e14092434e0e P 8c47b432cb734251bdded81a9470eda3: Deleting consensus metadata
I20260805 22:46:56.978251 6492 tablet_bootstrap.cc:492] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: Bootstrap starting.
I20260805 22:46:56.979730 6492 tablet_bootstrap.cc:654] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:56.981870 6493 tablet_bootstrap.cc:492] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: Bootstrap starting.
I20260805 22:46:56.982208 6531 raft_consensus.cc:2229] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:46:56.983210 6531 raft_consensus.cc:2258] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:46:56.983568 6493 tablet_bootstrap.cc:654] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:56.985189 6531 ts_tablet_manager.cc:1905] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:46:56.992828 6492 tablet_bootstrap.cc:492] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: No bootstrap required, opened a new log
I20260805 22:46:56.993556 6492 ts_tablet_manager.cc:1397] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20260805 22:46:56.993713 6531 ts_tablet_manager.cc:1918] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:46:56.993932 6531 log.cc:1196] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/8e6a1ba38afd48d6a802e14092434e0e
I20260805 22:46:56.993934 6530 tablet_replica.cc:330] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: stopping tablet replica
I20260805 22:46:56.994472 6530 raft_consensus.cc:2229] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:46:56.994686 6531 ts_tablet_manager.cc:1939] T 8e6a1ba38afd48d6a802e14092434e0e P c4790cecfade452c833d9062784835a7: Deleting consensus metadata
I20260805 22:46:56.994969 6492 raft_consensus.cc:348] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:56.995302 6492 raft_consensus.cc:374] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:56.995347 6530 raft_consensus.cc:2258] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:46:56.995528 6492 raft_consensus.cc:729] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4790cecfade452c833d9062784835a7, State: Initialized, Role: FOLLOWER
I20260805 22:46:56.995909 6492 consensus_queue.cc:260] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:56.997690 6492 ts_tablet_manager.cc:1428] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260805 22:46:56.998049 6530 ts_tablet_manager.cc:1905] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:46:56.999817 5745 catalog_manager.cc:4841] TS 8c47b432cb734251bdded81a9470eda3 (127.4.28.1:39219): tablet 8e6a1ba38afd48d6a802e14092434e0e (table test-table [id=e5d10c958a1342f3811aa854b80d4f95]) successfully deleted
I20260805 22:46:57.005303 6493 tablet_bootstrap.cc:492] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: No bootstrap required, opened a new log
I20260805 22:46:57.005635 6493 ts_tablet_manager.cc:1397] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.005s
I20260805 22:46:57.006601 6493 raft_consensus.cc:348] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.006809 6530 ts_tablet_manager.cc:1918] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:46:57.006947 6493 raft_consensus.cc:374] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:57.006923 5745 catalog_manager.cc:4841] TS c4790cecfade452c833d9062784835a7 (127.4.28.2:41441): tablet 8e6a1ba38afd48d6a802e14092434e0e (table test-table [id=e5d10c958a1342f3811aa854b80d4f95]) successfully deleted
I20260805 22:46:57.007121 6530 log.cc:1196] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/8e6a1ba38afd48d6a802e14092434e0e
I20260805 22:46:57.007112 6493 raft_consensus.cc:729] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c47b432cb734251bdded81a9470eda3, State: Initialized, Role: FOLLOWER
I20260805 22:46:57.008191 6530 ts_tablet_manager.cc:1939] T 8e6a1ba38afd48d6a802e14092434e0e P b8738a9446a149e1874dd162e326dee2: Deleting consensus metadata
I20260805 22:46:57.009176 6493 consensus_queue.cc:260] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.010424 5746 catalog_manager.cc:4841] TS b8738a9446a149e1874dd162e326dee2 (127.4.28.3:46577): tablet 8e6a1ba38afd48d6a802e14092434e0e (table test-table [id=e5d10c958a1342f3811aa854b80d4f95]) successfully deleted
I20260805 22:46:57.012264 6493 ts_tablet_manager.cc:1428] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260805 22:46:57.125103 6505 raft_consensus.cc:482] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:46:57.125370 6505 raft_consensus.cc:504] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.127086 6505 leader_election.cc:288] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4790cecfade452c833d9062784835a7 (127.4.28.2:41441), b8738a9446a149e1874dd162e326dee2 (127.4.28.3:46577)
I20260805 22:46:57.135136 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.128058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:47985 (local address 127.4.28.2:41441)
0805 22:46:57.128238 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.128243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.128476 (+ 233us) server_negotiation.cc:407] Connection header received
0805 22:46:57.128722 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.128726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.128804 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.128898 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:57.129635 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.130786 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.132714 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.133971 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.134160 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.134274 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.134648 (+ 374us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.134798 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:46:57.135335 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.127519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:55423 (local address 127.4.28.3:46577)
0805 22:46:57.127891 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.127898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.128227 (+ 329us) server_negotiation.cc:407] Connection header received
0805 22:46:57.128580 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.128587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.128670 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.128774 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:57.129408 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.130537 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.133161 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.134425 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.134545 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.134643 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.134949 (+ 306us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.135096 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260805 22:46:57.135452 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.127379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:46577 (local address 127.4.28.1:55423)
0805 22:46:57.128090 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:57.128112 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:57.128358 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:57.128808 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:57.128813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:57.128837 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:57.129228 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.129241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.130708 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.130714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.132979 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.132996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.134648 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.134656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.134795 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.134819 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 22:46:57.134903 (+ 84us) client_negotiation.cc:240] Negotiation successful
0805 22:46:57.135047 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":4,"thread_start_us":363,"threads_started":1}
I20260805 22:46:57.135977 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.127394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:41441 (local address 127.4.28.1:47985)
0805 22:46:57.128367 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:57.128388 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:57.128579 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:57.128976 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:57.128981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:57.129013 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:57.129452 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.129464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.130916 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.130930 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.132534 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.132551 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.134228 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.134235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.134429 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.134469 (+ 40us) client_negotiation.cc:769] Sending connection context
0805 22:46:57.135444 (+ 975us) client_negotiation.cc:240] Negotiation successful
0805 22:46:57.135586 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":32,"thread_start_us":883,"threads_started":1}
I20260805 22:46:57.136173 6387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7385cac5b23b4be5a80beee454424d63" candidate_uuid: "8c47b432cb734251bdded81a9470eda3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8738a9446a149e1874dd162e326dee2" is_pre_election: true
I20260805 22:46:57.136457 6387 raft_consensus.cc:2454] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c47b432cb734251bdded81a9470eda3 in term 0.
I20260805 22:46:57.136967 5895 leader_election.cc:302] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c47b432cb734251bdded81a9470eda3, b8738a9446a149e1874dd162e326dee2; no voters:
I20260805 22:46:57.136883 6184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7385cac5b23b4be5a80beee454424d63" candidate_uuid: "8c47b432cb734251bdded81a9470eda3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4790cecfade452c833d9062784835a7" is_pre_election: true
I20260805 22:46:57.137356 6505 raft_consensus.cc:2781] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:46:57.137373 6184 raft_consensus.cc:2454] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c47b432cb734251bdded81a9470eda3 in term 0.
I20260805 22:46:57.137480 6505 raft_consensus.cc:482] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:46:57.137607 6505 raft_consensus.cc:3037] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:57.139946 6505 raft_consensus.cc:504] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.140594 6505 leader_election.cc:288] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [CANDIDATE]: Term 1 election: Requested vote from peers c4790cecfade452c833d9062784835a7 (127.4.28.2:41441), b8738a9446a149e1874dd162e326dee2 (127.4.28.3:46577)
I20260805 22:46:57.141203 6387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7385cac5b23b4be5a80beee454424d63" candidate_uuid: "8c47b432cb734251bdded81a9470eda3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8738a9446a149e1874dd162e326dee2"
I20260805 22:46:57.141215 6184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7385cac5b23b4be5a80beee454424d63" candidate_uuid: "8c47b432cb734251bdded81a9470eda3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4790cecfade452c833d9062784835a7"
I20260805 22:46:57.141402 6387 raft_consensus.cc:3037] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:57.141440 6184 raft_consensus.cc:3037] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:57.143412 6387 raft_consensus.cc:2454] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c47b432cb734251bdded81a9470eda3 in term 1.
I20260805 22:46:57.143894 5895 leader_election.cc:302] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c47b432cb734251bdded81a9470eda3, b8738a9446a149e1874dd162e326dee2; no voters:
I20260805 22:46:57.144189 6505 raft_consensus.cc:2781] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:46:57.144682 6505 raft_consensus.cc:686] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 1 LEADER]: Becoming Leader. State: Replica: 8c47b432cb734251bdded81a9470eda3, State: Running, Role: LEADER
I20260805 22:46:57.144968 6184 raft_consensus.cc:2454] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c47b432cb734251bdded81a9470eda3 in term 1.
I20260805 22:46:57.145412 6505 consensus_queue.cc:237] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.149288 5769 catalog_manager.cc:5495] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c47b432cb734251bdded81a9470eda3 (127.4.28.1). New cstate: current_term: 1 leader_uuid: "8c47b432cb734251bdded81a9470eda3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } health_report { overall_health: UNKNOWN } } }
W20260805 22:46:57.183924 6060 tablet.cc:2369] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 05 22:46:57 dist-test-slave-7lbf krb5kdc[5709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970014, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:57.243343 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.228110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:36471 (local address 127.0.0.1:42690)
0805 22:46:57.228839 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:57.228864 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:57.229071 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:57.229452 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:57.229461 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:57.230310 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:46:57.230579 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.230591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.231751 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.231755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.232741 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.232756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.233540 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.233547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.233741 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.234444 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:46:57.234469 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:46:57.236984 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:46:57.240611 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:57.240632 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:57.240640 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:57.241118 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:57.241552 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:46:57.241558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:46:57.241562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:46:57.241709 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:46:57.242261 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:46:57.242271 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:46:57.242457 (+ 186us) client_negotiation.cc:769] Sending connection context
0805 22:46:57.242598 (+ 141us) client_negotiation.cc:240] Negotiation successful
0805 22:46:57.242897 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":383,"threads_started":1}
I20260805 22:46:57.243381 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.228221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.4.28.62:36471)
0805 22:46:57.229183 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.229188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.229209 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:46:57.229252 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.229256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.229308 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.229398 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:57.230753 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.231601 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.232898 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.233332 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.233464 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.237231 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:57.237237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:57.237241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:57.237276 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:57.240415 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.241260 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.241267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.241270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.241405 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.241822 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.241826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.241828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.242124 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:57.242262 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.242720 (+ 458us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.243037 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":833,"thread_start_us":701,"threads_started":1}
I20260805 22:46:57.249751 5769 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:42690:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260805 22:46:57.251188 5769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:46:57.259158 6149 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae027e6bce34c08916b02b652e0f922 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=556ab50480614723b3a17566ed93cf6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 22:46:57.259157 5965 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae027e6bce34c08916b02b652e0f922 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=556ab50480614723b3a17566ed93cf6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 22:46:57.259434 6356 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae027e6bce34c08916b02b652e0f922 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=556ab50480614723b3a17566ed93cf6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 22:46:57.259826 5965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae027e6bce34c08916b02b652e0f922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:57.259841 6149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae027e6bce34c08916b02b652e0f922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:57.260052 6356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae027e6bce34c08916b02b652e0f922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:57.269104 6491 tablet_bootstrap.cc:492] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: Bootstrap starting.
I20260805 22:46:57.269590 6492 tablet_bootstrap.cc:492] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: Bootstrap starting.
I20260805 22:46:57.271116 6491 tablet_bootstrap.cc:654] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:57.271258 6492 tablet_bootstrap.cc:654] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:57.272461 6493 tablet_bootstrap.cc:492] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: Bootstrap starting.
I20260805 22:46:57.274034 6491 tablet_bootstrap.cc:492] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: No bootstrap required, opened a new log
I20260805 22:46:57.274246 6491 ts_tablet_manager.cc:1397] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260805 22:46:57.274283 6493 tablet_bootstrap.cc:654] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:57.275128 6491 raft_consensus.cc:348] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.275409 6491 raft_consensus.cc:374] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:57.275502 6491 raft_consensus.cc:729] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8738a9446a149e1874dd162e326dee2, State: Initialized, Role: FOLLOWER
I20260805 22:46:57.276072 6491 consensus_queue.cc:260] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.277195 6500 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.277585 6500 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:46:57.278782 6491 ts_tablet_manager.cc:1428] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260805 22:46:57.278831 6493 tablet_bootstrap.cc:492] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: No bootstrap required, opened a new log
I20260805 22:46:57.279017 6493 ts_tablet_manager.cc:1397] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260805 22:46:57.279614 6493 raft_consensus.cc:348] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.279794 6493 raft_consensus.cc:374] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:57.279844 6493 raft_consensus.cc:729] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c47b432cb734251bdded81a9470eda3, State: Initialized, Role: FOLLOWER
I20260805 22:46:57.280074 6493 consensus_queue.cc:260] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.280525 6505 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.280844 6505 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:46:57.281375 6493 ts_tablet_manager.cc:1428] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260805 22:46:57.288084 6492 tablet_bootstrap.cc:492] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: No bootstrap required, opened a new log
I20260805 22:46:57.288282 6492 ts_tablet_manager.cc:1397] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260805 22:46:57.289021 6492 raft_consensus.cc:348] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.289234 6492 raft_consensus.cc:374] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:57.289302 6492 raft_consensus.cc:729] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4790cecfade452c833d9062784835a7, State: Initialized, Role: FOLLOWER
I20260805 22:46:57.289611 6492 consensus_queue.cc:260] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.290300 6492 ts_tablet_manager.cc:1428] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260805 22:46:57.290138 6522 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.290640 6522 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:46:57.333638 6500 raft_consensus.cc:482] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:46:57.333865 6500 raft_consensus.cc:504] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.335491 6500 leader_election.cc:288] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c47b432cb734251bdded81a9470eda3 (127.4.28.1:39219), c4790cecfade452c833d9062784835a7 (127.4.28.2:41441)
I20260805 22:46:57.344295 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.335990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:42699 (local address 127.4.28.1:39219)
0805 22:46:57.337040 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.337047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.337070 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:46:57.337169 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.337176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.337260 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.337924 (+ 664us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:57.338113 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.339201 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.341709 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.342959 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.343583 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.343693 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.343796 (+ 103us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.343946 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":1003,"threads_started":1}
I20260805 22:46:57.344352 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.335803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:39219 (local address 127.4.28.3:42699)
0805 22:46:57.336404 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:57.336424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:57.336559 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:57.337435 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:57.337440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:57.337471 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:57.337750 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.337763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.339364 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.339370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.341501 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.341549 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.343145 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.343151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.343622 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.343657 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:46:57.343771 (+ 114us) client_negotiation.cc:240] Negotiation successful
0805 22:46:57.343921 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":15,"thread_start_us":299,"threads_started":1}
I20260805 22:46:57.345265 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.336567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:41441 (local address 127.4.28.3:34363)
0805 22:46:57.337632 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:57.337655 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:57.337808 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:57.338211 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:57.338216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:57.338244 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:46:57.338544 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.338555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.339812 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.339828 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.341310 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.341327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.343345 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.343351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.344661 (+ 1310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.344701 (+ 40us) client_negotiation.cc:769] Sending connection context
0805 22:46:57.344855 (+ 154us) client_negotiation.cc:240] Negotiation successful
0805 22:46:57.344997 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":8,"thread_start_us":667,"threads_started":1}
I20260805 22:46:57.345172 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae027e6bce34c08916b02b652e0f922" candidate_uuid: "b8738a9446a149e1874dd162e326dee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c47b432cb734251bdded81a9470eda3" is_pre_election: true
I20260805 22:46:57.345465 5987 raft_consensus.cc:2454] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8738a9446a149e1874dd162e326dee2 in term 0.
I20260805 22:46:57.345996 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.336785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:34363 (local address 127.4.28.2:41441)
0805 22:46:57.337000 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.337006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.337772 (+ 766us) server_negotiation.cc:407] Connection header received
0805 22:46:57.337958 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.337963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.338073 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.338170 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:46:57.338743 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.339677 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.341505 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.342887 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.343035 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.343177 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.344918 (+ 1741us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.345733 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260805 22:46:57.346067 6285 leader_election.cc:302] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c47b432cb734251bdded81a9470eda3, b8738a9446a149e1874dd162e326dee2; no voters:
I20260805 22:46:57.346545 6500 raft_consensus.cc:2781] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:46:57.346563 6184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae027e6bce34c08916b02b652e0f922" candidate_uuid: "b8738a9446a149e1874dd162e326dee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4790cecfade452c833d9062784835a7" is_pre_election: true
I20260805 22:46:57.346676 6500 raft_consensus.cc:482] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:46:57.346758 6500 raft_consensus.cc:3037] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:57.346803 6184 raft_consensus.cc:2454] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8738a9446a149e1874dd162e326dee2 in term 0.
I20260805 22:46:57.350085 6500 raft_consensus.cc:504] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.350920 6500 leader_election.cc:288] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [CANDIDATE]: Term 1 election: Requested vote from peers 8c47b432cb734251bdded81a9470eda3 (127.4.28.1:39219), c4790cecfade452c833d9062784835a7 (127.4.28.2:41441)
I20260805 22:46:57.351637 6184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae027e6bce34c08916b02b652e0f922" candidate_uuid: "b8738a9446a149e1874dd162e326dee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4790cecfade452c833d9062784835a7"
I20260805 22:46:57.351648 5987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae027e6bce34c08916b02b652e0f922" candidate_uuid: "b8738a9446a149e1874dd162e326dee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c47b432cb734251bdded81a9470eda3"
I20260805 22:46:57.351897 6184 raft_consensus.cc:3037] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:57.351907 5987 raft_consensus.cc:3037] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:57.354465 5987 raft_consensus.cc:2454] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8738a9446a149e1874dd162e326dee2 in term 1.
I20260805 22:46:57.355031 6285 leader_election.cc:302] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c47b432cb734251bdded81a9470eda3, b8738a9446a149e1874dd162e326dee2; no voters:
I20260805 22:46:57.355053 6184 raft_consensus.cc:2454] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8738a9446a149e1874dd162e326dee2 in term 1.
I20260805 22:46:57.355441 6500 raft_consensus.cc:2781] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:46:57.356062 6500 raft_consensus.cc:686] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [term 1 LEADER]: Becoming Leader. State: Replica: b8738a9446a149e1874dd162e326dee2, State: Running, Role: LEADER
I20260805 22:46:57.356509 6500 consensus_queue.cc:237] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } }
I20260805 22:46:57.359057 6563 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: TxnStatusTablet state changed. Reason: New leader b8738a9446a149e1874dd162e326dee2. Latest consensus state: current_term: 1 leader_uuid: "b8738a9446a149e1874dd162e326dee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.359484 6563 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: This TxnStatusTablet replica's current role is: LEADER
I20260805 22:46:57.361272 5769 catalog_manager.cc:5495] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8738a9446a149e1874dd162e326dee2 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "b8738a9446a149e1874dd162e326dee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } health_report { overall_health: HEALTHY } } }
I20260805 22:46:57.361413 6566 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260805 22:46:57.361730 6566 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260805 22:46:57.400065 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.392144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.4.28.3:46577)
0805 22:46:57.392338 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.392344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.392509 (+ 165us) server_negotiation.cc:407] Connection header received
0805 22:46:57.392903 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.392913 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.393036 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.393217 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:46:57.393713 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.394911 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.396991 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.397781 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.397907 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.398535 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:46:57.399340 (+ 805us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:46:57.399480 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.399656 (+ 176us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.399746 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:46:57.400157 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.391979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:46577 (local address 127.0.0.1:43106)
0805 22:46:57.392402 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:57.392425 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:57.392624 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:57.393258 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:57.393264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:57.393278 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:46:57.393550 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.393566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.395102 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.395109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.396783 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.396806 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.397973 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.397980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.398200 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.398295 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:46:57.399470 (+ 1175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:46:57.399511 (+ 41us) client_negotiation.cc:769] Sending connection context
0805 22:46:57.399614 (+ 103us) client_negotiation.cc:240] Negotiation successful
0805 22:46:57.399808 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260805 22:46:57.405788 5987 raft_consensus.cc:1261] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3 [term 1 FOLLOWER]: Refusing update from remote peer b8738a9446a149e1874dd162e326dee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:46:57.406023 6184 raft_consensus.cc:1261] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7 [term 1 FOLLOWER]: Refusing update from remote peer b8738a9446a149e1874dd162e326dee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:46:57.406708 6500 consensus_queue.cc:1035] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:46:57.407096 6563 consensus_queue.cc:1035] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:46:57.419276 6505 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: TxnStatusTablet state changed. Reason: New leader b8738a9446a149e1874dd162e326dee2. Latest consensus state: current_term: 1 leader_uuid: "b8738a9446a149e1874dd162e326dee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.419644 6505 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:46:57.422294 6518 mvcc.cc:205] Tried to move back new op lower bound from 7315333191285428224 to 7315333191097020416. Current Snapshot: MvccSnapshot[applied={T|T < 7315333191285428224}]
I20260805 22:46:57.423672 6570 mvcc.cc:205] Tried to move back new op lower bound from 7315333191285428224 to 7315333191097020416. Current Snapshot: MvccSnapshot[applied={T|T < 7315333191285428224}]
I20260805 22:46:57.424008 6522 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: TxnStatusTablet state changed. Reason: New leader b8738a9446a149e1874dd162e326dee2. Latest consensus state: current_term: 1 leader_uuid: "b8738a9446a149e1874dd162e326dee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.424300 6522 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:46:57.431969 6563 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8738a9446a149e1874dd162e326dee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.437623 6563 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: This TxnStatusTablet replica's current role is: LEADER
I20260805 22:46:57.439347 6563 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8738a9446a149e1874dd162e326dee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.441426 6563 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P b8738a9446a149e1874dd162e326dee2: This TxnStatusTablet replica's current role is: LEADER
I20260805 22:46:57.448686 6505 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8738a9446a149e1874dd162e326dee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.449119 6505 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P 8c47b432cb734251bdded81a9470eda3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:46:57.463732 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.436334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.4.28.1:39219)
0805 22:46:57.436576 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.436583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.453677 (+ 17094us) server_negotiation.cc:407] Connection header received
0805 22:46:57.455184 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.455192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.455357 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.455516 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:46:57.456492 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.457735 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.460147 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.460911 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.461036 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.461888 (+ 852us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:46:57.462800 (+ 912us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:46:57.462947 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.463259 (+ 312us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.463376 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:46:57.465974 6522 tablet_replica.cc:439] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8738a9446a149e1874dd162e326dee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c47b432cb734251bdded81a9470eda3" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39219 } } peers { permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 } } peers { permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 } } }
I20260805 22:46:57.466275 6522 tablet_replica.cc:442] T 4ae027e6bce34c08916b02b652e0f922 P c4790cecfade452c833d9062784835a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:46:57.484386 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.478648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:55089 (local address 127.4.28.3:46577)
0805 22:46:57.478846 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.478853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.479277 (+ 424us) server_negotiation.cc:407] Connection header received
0805 22:46:57.479479 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.479484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.479557 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.479634 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:46:57.480168 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.480958 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.482512 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.483013 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.483132 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.483625 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:46:57.483745 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:46:57.483835 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.483971 (+ 136us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.484067 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 22:46:57.484464 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.478559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:46577 (local address 127.4.28.1:55089)
0805 22:46:57.479156 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:57.479173 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:57.479338 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:57.479671 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:57.479675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:57.479687 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:46:57.479994 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.480003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.481144 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.481148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.482359 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.482374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.483210 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.483215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.483348 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.483413 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:46:57.483849 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:46:57.483862 (+ 13us) client_negotiation.cc:769] Sending connection context
0805 22:46:57.483932 (+ 70us) client_negotiation.cc:240] Negotiation successful
0805 22:46:57.484093 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":299,"threads_started":1}
I20260805 22:46:57.496853 6184 raft_consensus.cc:1261] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c47b432cb734251bdded81a9470eda3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:46:57.496899 6387 raft_consensus.cc:1261] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Refusing update from remote peer 8c47b432cb734251bdded81a9470eda3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:46:57.497619 6541 consensus_queue.cc:1035] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8738a9446a149e1874dd162e326dee2" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:46:57.497997 6505 consensus_queue.cc:1035] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4790cecfade452c833d9062784835a7" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:46:57.564404 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.556607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:39561 (local address 127.4.28.1:39219)
0805 22:46:57.556810 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.556817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.557664 (+ 847us) server_negotiation.cc:407] Connection header received
0805 22:46:57.558075 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.558081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.558193 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.558263 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:46:57.559149 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.560196 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.562096 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.562716 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.562826 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.563529 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:46:57.563680 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:46:57.563742 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.563985 (+ 243us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.564080 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:46:57.565086 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.556627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:39219 (local address 127.4.28.3:39561)
0805 22:46:57.557474 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0805 22:46:57.557500 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0805 22:46:57.557859 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:46:57.558513 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:46:57.558518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:46:57.558530 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:46:57.558923 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.558937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.560342 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.560345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.561924 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:46:57.561943 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.563028 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.563035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:46:57.563217 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.563301 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:46:57.563822 (+ 521us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:46:57.563842 (+ 20us) client_negotiation.cc:769] Sending connection context
0805 22:46:57.564618 (+ 776us) client_negotiation.cc:240] Negotiation successful
0805 22:46:57.564767 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":3181,"threads_started":1}
I20260805 22:46:57.618091 5768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:46:57.618386 5768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:46:57.622457 5768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ef4ebacfd83483d899e83868b9ec519: Sending DeleteTablet for 3 replicas of tablet 7385cac5b23b4be5a80beee454424d63
I20260805 22:46:57.623538 6149 tablet_service.cc:1514] Processing DeleteTablet for tablet 7385cac5b23b4be5a80beee454424d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:46:57 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:32928
I20260805 22:46:57.623600 5965 tablet_service.cc:1514] Processing DeleteTablet for tablet 7385cac5b23b4be5a80beee454424d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:46:57 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:33782
I20260805 22:46:57.625767 6356 tablet_service.cc:1514] Processing DeleteTablet for tablet 7385cac5b23b4be5a80beee454424d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:46:57 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:43092
I20260805 22:46:57.638435 6597 tablet_replica.cc:330] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: stopping tablet replica
I20260805 22:46:57.638854 6597 raft_consensus.cc:2229] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:46:57.639595 6597 raft_consensus.cc:2258] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:46:57.640933 6597 ts_tablet_manager.cc:1905] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:46:57.644765 6599 tablet_replica.cc:330] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: stopping tablet replica
I20260805 22:46:57.645231 6599 raft_consensus.cc:2229] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:46:57.645720 6599 raft_consensus.cc:2258] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:46:57.647128 6599 ts_tablet_manager.cc:1905] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:46:57.647641 6596 tablet_replica.cc:330] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: stopping tablet replica
I20260805 22:46:57.647934 6596 raft_consensus.cc:2229] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:46:57.648386 6596 raft_consensus.cc:2258] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:46:57.650142 6597 ts_tablet_manager.cc:1918] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:46:57.650369 6597 log.cc:1196] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/7385cac5b23b4be5a80beee454424d63
I20260805 22:46:57.650966 6597 ts_tablet_manager.cc:1939] T 7385cac5b23b4be5a80beee454424d63 P 8c47b432cb734251bdded81a9470eda3: Deleting consensus metadata
I20260805 22:46:57.652350 5745 catalog_manager.cc:4841] TS 8c47b432cb734251bdded81a9470eda3 (127.4.28.1:39219): tablet 7385cac5b23b4be5a80beee454424d63 (table test-table [id=106909d703504f50be5bf535760afb09]) successfully deleted
I20260805 22:46:57.654158 6596 ts_tablet_manager.cc:1905] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:46:57.659387 6599 ts_tablet_manager.cc:1918] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:46:57.659643 6599 log.cc:1196] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/7385cac5b23b4be5a80beee454424d63
I20260805 22:46:57.660180 6599 ts_tablet_manager.cc:1939] T 7385cac5b23b4be5a80beee454424d63 P b8738a9446a149e1874dd162e326dee2: Deleting consensus metadata
I20260805 22:46:57.661692 5746 catalog_manager.cc:4841] TS b8738a9446a149e1874dd162e326dee2 (127.4.28.3:46577): tablet 7385cac5b23b4be5a80beee454424d63 (table test-table [id=106909d703504f50be5bf535760afb09]) successfully deleted
I20260805 22:46:57.667748 6596 ts_tablet_manager.cc:1918] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:46:57.668057 6596 log.cc:1196] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/7385cac5b23b4be5a80beee454424d63
I20260805 22:46:57.668619 6596 ts_tablet_manager.cc:1939] T 7385cac5b23b4be5a80beee454424d63 P c4790cecfade452c833d9062784835a7: Deleting consensus metadata
I20260805 22:46:57.670034 5745 catalog_manager.cc:4841] TS c4790cecfade452c833d9062784835a7 (127.4.28.2:41441): tablet 7385cac5b23b4be5a80beee454424d63 (table test-table [id=106909d703504f50be5bf535760afb09]) successfully deleted
Aug 05 22:46:57 dist-test-slave-7lbf krb5kdc[5709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970016, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:46:57.684207 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.655029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.4.28.1:39219)
0805 22:46:57.655225 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.655232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.662277 (+ 7045us) server_negotiation.cc:407] Connection header received
0805 22:46:57.662611 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.662619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.662709 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.662790 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:57.665135 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.666477 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.668495 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.669229 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.669360 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.675489 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:57.675515 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:57.675540 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:57.675609 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:57.680167 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.681343 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.681356 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.681360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.681669 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.682196 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.682202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.682205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.682816 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:57.683150 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.683541 (+ 391us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.683809 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260805 22:46:57.685295 6053 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44312
I20260805 22:46:57.707866 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.689806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.4.28.62:36471)
0805 22:46:57.690122 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.690129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.690670 (+ 541us) server_negotiation.cc:407] Connection header received
0805 22:46:57.690861 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.690868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.690960 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.691072 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:57.692963 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.694252 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.696206 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.696903 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.697056 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.700357 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:57.700365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:57.700370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:57.700428 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:57.704283 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.705301 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.705308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.705312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.705564 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.706111 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.706116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.706120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.706550 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:57.706837 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.707218 (+ 381us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.707507 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
W20260805 22:46:57.708606 5768 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33748
I20260805 22:46:57.712144 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 5871
I20260805 22:46:57.736137 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.714246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.4.28.1:39219)
0805 22:46:57.714449 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.714455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.718153 (+ 3698us) server_negotiation.cc:407] Connection header received
0805 22:46:57.718389 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.718396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.718538 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.718662 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:57.720347 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.721697 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.723641 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.724459 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.724595 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.727752 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:57.727760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:57.727765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:57.727820 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:57.731880 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.732938 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.732946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.732949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.733178 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.733839 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.733846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.733849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.734294 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:57.734597 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.735055 (+ 458us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.735337 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260805 22:46:57.737102 6053 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44328
W20260805 22:46:57.737844 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 5871: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:46:57.738129 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 5871
I20260805 22:46:57.762921 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 6063
Aug 05 22:46:57 dist-test-slave-7lbf krb5kdc[5709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970016, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:46:57.782285 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.763909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.4.28.2:41441)
0805 22:46:57.764136 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.764142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.764166 (+ 24us) server_negotiation.cc:407] Connection header received
0805 22:46:57.764240 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.764245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.764354 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.764480 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:57.765817 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.766980 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.768346 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.769060 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.769233 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.774098 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:57.774122 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:57.774145 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:57.774200 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:57.778452 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.779576 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.779589 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.779593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.779797 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.780337 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.780342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.780346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.780869 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:57.781227 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.781627 (+ 400us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.781895 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260805 22:46:57.783488 6250 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37476
W20260805 22:46:57.784332 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 6063: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:46:57.784543 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 6063
I20260805 22:46:57.832717 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 6263
Aug 05 22:46:57 dist-test-slave-7lbf krb5kdc[5709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970016, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:46:57.851840 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:57.833703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.4.28.3:46577)
0805 22:46:57.833958 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:57.833965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:57.833988 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:46:57.834193 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:57.834200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:57.834308 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:57.834462 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:57.836254 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.837153 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.839047 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:57.839512 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:57.839645 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:57.845289 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:57.845307 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:57.845326 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:57.845387 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:57.848646 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.849470 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.849478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.849481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.849697 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:57.850120 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:57.850124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:57.850126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:57.850530 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:57.850915 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:57.851227 (+ 312us) server_negotiation.cc:299] Negotiation successful
0805 22:46:57.851477 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260805 22:46:57.852603 6455 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51656
W20260805 22:46:57.853406 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 6263: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:46:57.853619 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 6263
I20260805 22:46:57.884256 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 5724
I20260805 22:46:57.885646 5824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:46:58.005345 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 5724
2026-08-05T22:46:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6383 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:46:58 dist-test-slave-7lbf krb5kdc[6610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:46:58 dist-test-slave-7lbf krb5kdc[6610](info): set up 2 sockets
Aug 05 22:46:58 dist-test-slave-7lbf krb5kdc[6610](info): commencing operation
krb5kdc: starting...
W20260805 22:46:59.189111 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:46:59 dist-test-slave-7lbf krb5kdc[6610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:46:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:46:59.429888 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:40619
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:40619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:46:59.781826 6628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:46:59.782167 6628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:46:59.782307 6628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:46:59.789973 6628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:46:59.790112 6628 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260805 22:46:59.790154 6628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:46:59.790251 6628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:46:59.790321 6628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:46:59.790413 6628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:46:59.800720 6628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:40619
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:46:59.802238 6628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:46:59.803673 6628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:46:59.816401 6634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:59.816371 6638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:46:59.816431 6635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:46:59.817317 6628 server_base.cc:1034] running on GCE node
I20260805 22:46:59.818267 6628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:46:59.819919 6628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:46:59.821162 6628 hybrid_clock.cc:648] HybridClock initialized: now 1785970019821085 us; error 57 us; skew 500 ppm
I20260805 22:46:59.821588 6628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:46:59 dist-test-slave-7lbf krb5kdc[6610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970019, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:46:59.829996 6628 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:46:59.831846 6628 webserver.cc:460] Webserver started at http://127.4.28.62:41531/ using document root <none> and password file <none>
I20260805 22:46:59.832674 6628 fs_manager.cc:359] Metadata directory not provided
I20260805 22:46:59.832795 6628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:46:59.833155 6628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:46:59.836512 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "352b2b1af1174c948aba0d7076f3514f"
format_stamp: "Formatted at 2026-08-05 22:46:59 on dist-test-slave-7lbf"
server_key: "973a407c9e40c71bf853069ddc762f7b"
server_key_iv: "5fbead3a24efa00a833c6f3014ce084c"
server_key_version: "encryptionkey@0"
I20260805 22:46:59.837404 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "352b2b1af1174c948aba0d7076f3514f"
format_stamp: "Formatted at 2026-08-05 22:46:59 on dist-test-slave-7lbf"
server_key: "973a407c9e40c71bf853069ddc762f7b"
server_key_iv: "5fbead3a24efa00a833c6f3014ce084c"
server_key_version: "encryptionkey@0"
I20260805 22:46:59.843364 6628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260805 22:46:59.847373 6646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:46:59.848934 6628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 22:46:59.849154 6628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "352b2b1af1174c948aba0d7076f3514f"
format_stamp: "Formatted at 2026-08-05 22:46:59 on dist-test-slave-7lbf"
server_key: "973a407c9e40c71bf853069ddc762f7b"
server_key_iv: "5fbead3a24efa00a833c6f3014ce084c"
server_key_version: "encryptionkey@0"
I20260805 22:46:59.849390 6628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:46:59.876744 6628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:46:59.881251 6628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:46:59.881573 6628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:46:59.903833 6628 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:40619
I20260805 22:46:59.903851 6733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:40619 every 8 connection(s)
I20260805 22:46:59.905349 6628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:46:59.909611 6734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:46:59.914119 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 6628
I20260805 22:46:59.914387 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:46:59.915221 4208 external_mini_cluster.cc:1428] Setting key bd106a56b46aed31d2792cb7f65c0551
I20260805 22:46:59.922472 6734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f: Bootstrap starting.
I20260805 22:46:59.926034 6734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f: Neither blocks nor log segments found. Creating new log.
I20260805 22:46:59.927235 6734 log.cc:824] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f: Log is configured to *not* fsync() on all Append() calls
I20260805 22:46:59.930703 6734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f: No bootstrap required, opened a new log
Aug 05 22:46:59 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:46:59.939952 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:46:59.917729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.4.28.62:40619)
0805 22:46:59.918588 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0805 22:46:59.918602 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:46:59.918651 (+ 49us) server_negotiation.cc:407] Connection header received
0805 22:46:59.920881 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:46:59.920912 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:46:59.921388 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:46:59.921865 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:46:59.923543 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:59.924955 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:59.926410 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:46:59.927206 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:46:59.927431 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:46:59.931884 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:46:59.931894 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:46:59.931913 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:46:59.931965 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:46:59.935193 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:59.936164 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:59.936171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:59.936179 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:59.936383 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:46:59.936830 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:46:59.936834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:46:59.936836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:46:59.937492 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:46:59.937809 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0805 22:46:59.938314 (+ 505us) server_negotiation.cc:299] Negotiation successful
0805 22:46:59.938779 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":369,"threads_started":1}
I20260805 22:46:59.939951 6734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352b2b1af1174c948aba0d7076f3514f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40619 } }
I20260805 22:46:59.940435 6734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:46:59.940548 6734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352b2b1af1174c948aba0d7076f3514f, State: Initialized, Role: FOLLOWER
I20260805 22:46:59.941159 6734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352b2b1af1174c948aba0d7076f3514f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40619 } }
I20260805 22:46:59.941421 6734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:46:59.941576 6734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:46:59.941785 6734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:46:59.944204 6734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352b2b1af1174c948aba0d7076f3514f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40619 } }
I20260805 22:46:59.944762 6734 leader_election.cc:302] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 352b2b1af1174c948aba0d7076f3514f; no voters:
I20260805 22:46:59.945339 6734 leader_election.cc:288] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:46:59.945475 6743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:46:59.946244 6743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [term 1 LEADER]: Becoming Leader. State: Replica: 352b2b1af1174c948aba0d7076f3514f, State: Running, Role: LEADER
I20260805 22:46:59.946878 6743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352b2b1af1174c948aba0d7076f3514f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40619 } }
I20260805 22:46:59.947511 6734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:46:59.950413 6744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "352b2b1af1174c948aba0d7076f3514f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352b2b1af1174c948aba0d7076f3514f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40619 } } }
I20260805 22:46:59.950896 6744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [sys.catalog]: This master's current role is: LEADER
I20260805 22:46:59.950400 6746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 352b2b1af1174c948aba0d7076f3514f. Latest consensus state: current_term: 1 leader_uuid: "352b2b1af1174c948aba0d7076f3514f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352b2b1af1174c948aba0d7076f3514f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40619 } } }
I20260805 22:46:59.951189 6746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f [sys.catalog]: This master's current role is: LEADER
I20260805 22:46:59.952793 6752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:46:59.960836 6752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:46:59.971347 6752 catalog_manager.cc:1347] Generated new cluster ID: e6e313c48da7464ca2880224766ee7b7
I20260805 22:46:59.971520 6752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:46:59.983702 6752 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:46:59.985180 6752 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:46:59.991918 6752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f: Generated new TSK 0
I20260805 22:46:59.992841 6752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:47:00.053532 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40619
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:00.432798 6771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:00.433113 6771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:00.433223 6771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:00.440852 6771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:00.440981 6771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:00.441071 6771 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260805 22:47:00.441123 6771 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 22:47:00.441193 6771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:00.452214 6771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.28.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:00.453888 6771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:00.455464 6771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:00.469959 6778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:00.469985 6777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:00.470052 6782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:00.475525 6771 server_base.cc:1034] running on GCE node
I20260805 22:47:00.476305 6771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:00.477437 6771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:00.478786 6771 hybrid_clock.cc:648] HybridClock initialized: now 1785970020478738 us; error 47 us; skew 500 ppm
I20260805 22:47:00.479163 6771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:00 dist-test-slave-7lbf krb5kdc[6610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970020, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:00.482769 6771 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:47:00.484689 6771 webserver.cc:460] Webserver started at http://127.4.28.1:37417/ using document root <none> and password file <none>
I20260805 22:47:00.485554 6771 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:00.485709 6771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:00.486078 6771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:00.489430 6771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "d58d6a5391754278b2c2dfa2537d9e2c"
format_stamp: "Formatted at 2026-08-05 22:47:00 on dist-test-slave-7lbf"
server_key: "82ba9abb949416bb71ee55dcded24c56"
server_key_iv: "89a1bc3e95b640a99639b9847e12da0a"
server_key_version: "encryptionkey@0"
I20260805 22:47:00.490413 6771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "d58d6a5391754278b2c2dfa2537d9e2c"
format_stamp: "Formatted at 2026-08-05 22:47:00 on dist-test-slave-7lbf"
server_key: "82ba9abb949416bb71ee55dcded24c56"
server_key_iv: "89a1bc3e95b640a99639b9847e12da0a"
server_key_version: "encryptionkey@0"
I20260805 22:47:00.496979 6771 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260805 22:47:00.501240 6790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:00.502893 6771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:00.503103 6771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "d58d6a5391754278b2c2dfa2537d9e2c"
format_stamp: "Formatted at 2026-08-05 22:47:00 on dist-test-slave-7lbf"
server_key: "82ba9abb949416bb71ee55dcded24c56"
server_key_iv: "89a1bc3e95b640a99639b9847e12da0a"
server_key_version: "encryptionkey@0"
I20260805 22:47:00.503337 6771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:00.544643 6771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:00.549484 6771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:00.549826 6771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:00.552946 6771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:00.553104 6771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:00.553267 6771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:00.553359 6771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:47:00 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970020, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:00.581097 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:00.553849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:41235 (local address 127.4.28.62:40619)
0805 22:47:00.560466 (+ 6617us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:00.560483 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:00.560504 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:47:00.560577 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:00.560582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:00.560682 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:00.560816 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:00.563503 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:00.564783 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:00.567343 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:00.568030 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:00.568181 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:00.573705 (+ 5524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:00.573714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:00.573720 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:00.573772 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:00.577005 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:00.578292 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:00.578299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:00.578304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:00.578451 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:00.579106 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:00.579110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:00.579113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:00.579401 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:00.579552 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:00.580434 (+ 882us) server_negotiation.cc:299] Negotiation successful
0805 22:47:00.580708 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6455,"thread_start_us":6317,"threads_started":1}
I20260805 22:47:00.582702 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:00.555100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40619 (local address 127.4.28.1:41235)
0805 22:47:00.556448 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:00.556555 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:00.559555 (+ 3000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:00.561148 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:00.561158 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:00.562222 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:00.563170 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:00.563213 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:00.565072 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:00.565077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:00.567055 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:00.567072 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:00.568348 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:00.568357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:00.568687 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:00.569447 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:00.569499 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:00.573261 (+ 3762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:00.577220 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:00.577233 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:00.577262 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:00.578125 (+ 863us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:00.578573 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:00.578582 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:00.578586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:00.578922 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:00.579570 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:00.579580 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:00.580005 (+ 425us) client_negotiation.cc:769] Sending connection context
0805 22:47:00.580288 (+ 283us) client_negotiation.cc:240] Negotiation successful
0805 22:47:00.580920 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":453,"threads_started":1}
I20260805 22:47:00.604373 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:00.589769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:37193 (local address 127.4.28.62:40619)
0805 22:47:00.590021 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:00.590029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:00.590577 (+ 548us) server_negotiation.cc:407] Connection header received
0805 22:47:00.590792 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:00.590796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:00.590866 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:00.590943 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:00.592265 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:00.593049 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:00.595312 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:00.595773 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:00.595898 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:00.597957 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:00.597962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:00.597966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:00.598003 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:00.601084 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:00.602126 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:00.602131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:00.602134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:00.602334 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:00.602875 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:00.602879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:00.602881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:00.603236 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:00.603367 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:00.603905 (+ 538us) server_negotiation.cc:299] Negotiation successful
0805 22:47:00.604105 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260805 22:47:00.604777 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:00.589655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40619 (local address 127.4.28.1:37193)
0805 22:47:00.590420 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:00.590445 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:00.590657 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:00.590995 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:00.591000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:00.591667 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:00.592104 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:00.592120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:00.593223 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:00.593229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:00.595124 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:00.595145 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:00.595986 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:00.595993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:00.596173 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:00.596806 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:00.596825 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:00.597724 (+ 899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:00.601265 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:00.601272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:00.601276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:00.601967 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:00.602529 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:00.602536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:00.602540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:00.602746 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:00.603368 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:00.603375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:00.603631 (+ 256us) client_negotiation.cc:769] Sending connection context
0805 22:47:00.603804 (+ 173us) client_negotiation.cc:240] Negotiation successful
0805 22:47:00.604229 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":381,"threads_started":1}
I20260805 22:47:00.609992 6771 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:37413
I20260805 22:47:00.610021 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:37413 every 8 connection(s)
I20260805 22:47:00.611357 6771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:00.613714 6955 heartbeater.cc:347] Connected to a master server at 127.4.28.62:40619
I20260805 22:47:00.613794 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 6771
I20260805 22:47:00.614017 6955 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:00.614043 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:00.614863 4208 external_mini_cluster.cc:1428] Setting key a890b091bebe3c915bc47ff6f4f8667c
I20260805 22:47:00.614974 6955 heartbeater.cc:511] Master 127.4.28.62:40619 requested a full tablet report, sending...
I20260805 22:47:00.617502 6673 ts_manager.cc:194] Registered new tserver with Master: d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413)
I20260805 22:47:00.619112 6673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:41235
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:47:00.668721 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40619
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:01.030627 6961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:01.031049 6961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:01.031216 6961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:01.038873 6961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:01.039036 6961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:01.039165 6961 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260805 22:47:01.039238 6961 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 22:47:01.039348 6961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:01.049795 6961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.28.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:01.051366 6961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:01.052872 6961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:01.067049 6967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:01.067050 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:01.067276 6971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:01.068239 6961 server_base.cc:1034] running on GCE node
I20260805 22:47:01.068892 6961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:01.069942 6961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:01.071197 6961 hybrid_clock.cc:648] HybridClock initialized: now 1785970021071135 us; error 46 us; skew 500 ppm
I20260805 22:47:01.071579 6961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:01 dist-test-slave-7lbf krb5kdc[6610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:01.075006 6961 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:47:01.076640 6961 webserver.cc:460] Webserver started at http://127.4.28.2:44809/ using document root <none> and password file <none>
I20260805 22:47:01.077396 6961 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:01.077564 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:01.077924 6961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:01.081151 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "52f3ba3f6ee74de98c917371d059be80"
format_stamp: "Formatted at 2026-08-05 22:47:01 on dist-test-slave-7lbf"
server_key: "38ffd094e73eca24a0c133a63ccc29e9"
server_key_iv: "bee614a2466752f5848875100879acd1"
server_key_version: "encryptionkey@0"
I20260805 22:47:01.082015 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "52f3ba3f6ee74de98c917371d059be80"
format_stamp: "Formatted at 2026-08-05 22:47:01 on dist-test-slave-7lbf"
server_key: "38ffd094e73eca24a0c133a63ccc29e9"
server_key_iv: "bee614a2466752f5848875100879acd1"
server_key_version: "encryptionkey@0"
I20260805 22:47:01.087904 6961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260805 22:47:01.092064 6985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:01.093654 6961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:01.093847 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "52f3ba3f6ee74de98c917371d059be80"
format_stamp: "Formatted at 2026-08-05 22:47:01 on dist-test-slave-7lbf"
server_key: "38ffd094e73eca24a0c133a63ccc29e9"
server_key_iv: "bee614a2466752f5848875100879acd1"
server_key_version: "encryptionkey@0"
I20260805 22:47:01.094058 6961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:01.110904 6961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:01.115676 6961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:01.115995 6961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:01.119128 6961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:01.119251 6961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:01.119378 6961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:01.119475 6961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:47:01 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:01.144306 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.119694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:51303 (local address 127.4.28.62:40619)
0805 22:47:01.120488 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:01.120495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:01.121675 (+ 1180us) server_negotiation.cc:407] Connection header received
0805 22:47:01.123919 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:01.123925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:01.124017 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:01.124117 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:01.126693 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.127686 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.130697 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.131158 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.131298 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.137185 (+ 5887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:01.137194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:01.137199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:01.137252 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:01.140663 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.141613 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.141618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.141620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.141766 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.142292 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.142297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.142299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.142628 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:01.142780 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:01.143650 (+ 870us) server_negotiation.cc:299] Negotiation successful
0805 22:47:01.143913 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":516,"threads_started":1}
I20260805 22:47:01.145884 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.120430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40619 (local address 127.4.28.2:51303)
0805 22:47:01.121429 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:01.121487 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:01.123577 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:01.124462 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:01.124479 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:01.125352 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:01.126440 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.126467 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.127871 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.127877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.130431 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.130459 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.131480 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.131494 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.131903 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.132881 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:01.132910 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:01.136785 (+ 3875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:01.140853 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:01.140865 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:01.140888 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:01.141416 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:01.141894 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:01.141903 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:01.141907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:01.142154 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:01.142786 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:01.142796 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:01.143226 (+ 430us) client_negotiation.cc:769] Sending connection context
0805 22:47:01.143497 (+ 271us) client_negotiation.cc:240] Negotiation successful
0805 22:47:01.144136 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":399,"threads_started":1}
I20260805 22:47:01.171486 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.153798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:49593 (local address 127.4.28.62:40619)
0805 22:47:01.154028 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:01.154036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:01.154734 (+ 698us) server_negotiation.cc:407] Connection header received
0805 22:47:01.155073 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:01.155078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:01.155155 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:01.155248 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:01.156980 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.157933 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.159989 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.160606 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.160741 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.163484 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:01.163493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:01.163499 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:01.163551 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:01.167773 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.168759 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.168767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.168772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.169000 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.169478 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.169483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.169487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.170074 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:01.170269 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:01.170838 (+ 569us) server_negotiation.cc:299] Negotiation successful
0805 22:47:01.171095 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260805 22:47:01.171598 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.153753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40619 (local address 127.4.28.2:49593)
0805 22:47:01.154592 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:01.154617 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:01.154884 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:01.155316 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:01.155322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:01.156049 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:01.156723 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.156751 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.158101 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.158109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.159807 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.159832 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.161070 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.161077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.161212 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.162061 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:01.162081 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:01.163203 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:01.168009 (+ 4806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:01.168016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:01.168021 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:01.168536 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:01.169182 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:01.169186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:01.169192 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:01.169356 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:01.170237 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:01.170243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:01.170524 (+ 281us) client_negotiation.cc:769] Sending connection context
0805 22:47:01.170700 (+ 176us) client_negotiation.cc:240] Negotiation successful
0805 22:47:01.171118 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":389,"threads_started":1}
I20260805 22:47:01.176523 6961 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:34495
I20260805 22:47:01.176542 7158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:34495 every 8 connection(s)
I20260805 22:47:01.177928 6961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:01.180095 7159 heartbeater.cc:347] Connected to a master server at 127.4.28.62:40619
I20260805 22:47:01.180330 7159 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:01.181032 7159 heartbeater.cc:511] Master 127.4.28.62:40619 requested a full tablet report, sending...
I20260805 22:47:01.182525 6673 ts_manager.cc:194] Registered new tserver with Master: 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495)
I20260805 22:47:01.183336 6673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:51303
I20260805 22:47:01.184019 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 6961
I20260805 22:47:01.184224 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:47:01.184661 4208 external_mini_cluster.cc:1428] Setting key 12d5fabecd14e00e8aeb198c16e603c3
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:47:01.236589 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40619
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:01.589881 7166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:01.590204 7166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:01.590350 7166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:01.597930 7166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:01.598070 7166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:01.598183 7166 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260805 22:47:01.598245 7166 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 22:47:01.598322 7166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:01.608677 7166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.28.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:01.610272 7166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:01.611708 7166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 22:47:01.622583 6955 heartbeater.cc:503] Master 127.4.28.62:40619 was elected leader, sending a full tablet report...
W20260805 22:47:01.626299 7174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:01.626317 7173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:01.626379 7176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:01.626950 7166 server_base.cc:1034] running on GCE node
I20260805 22:47:01.627597 7166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:01.628594 7166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:01.629812 7166 hybrid_clock.cc:648] HybridClock initialized: now 1785970021629743 us; error 61 us; skew 500 ppm
I20260805 22:47:01.630216 7166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:01 dist-test-slave-7lbf krb5kdc[6610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:01.633533 7166 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:47:01.635169 7166 webserver.cc:460] Webserver started at http://127.4.28.3:45929/ using document root <none> and password file <none>
I20260805 22:47:01.635921 7166 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:01.636055 7166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:01.636372 7166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:01.639593 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "a69d11b65474431c82f000b800174059"
format_stamp: "Formatted at 2026-08-05 22:47:01 on dist-test-slave-7lbf"
server_key: "f7c6448d21938364437686e146e7a300"
server_key_iv: "529c48220f01524a14087a9c6f062f31"
server_key_version: "encryptionkey@0"
I20260805 22:47:01.640398 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "a69d11b65474431c82f000b800174059"
format_stamp: "Formatted at 2026-08-05 22:47:01 on dist-test-slave-7lbf"
server_key: "f7c6448d21938364437686e146e7a300"
server_key_iv: "529c48220f01524a14087a9c6f062f31"
server_key_version: "encryptionkey@0"
I20260805 22:47:01.646263 7166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260805 22:47:01.650804 7187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:01.652400 7166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:01.652596 7166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "a69d11b65474431c82f000b800174059"
format_stamp: "Formatted at 2026-08-05 22:47:01 on dist-test-slave-7lbf"
server_key: "f7c6448d21938364437686e146e7a300"
server_key_iv: "529c48220f01524a14087a9c6f062f31"
server_key_version: "encryptionkey@0"
I20260805 22:47:01.652838 7166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:01.678926 7166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:01.683990 7166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:01.684303 7166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:01.687317 7166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:01.687460 7166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:01.687593 7166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:01.687680 7166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:47:01 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:01.716037 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.688240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:58315 (local address 127.4.28.62:40619)
0805 22:47:01.688723 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:01.688728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:01.690175 (+ 1447us) server_negotiation.cc:407] Connection header received
0805 22:47:01.692625 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:01.692633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:01.692731 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:01.692822 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:01.695592 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.696847 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.699845 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.700570 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.700731 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.707193 (+ 6462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:01.707202 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:01.707208 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:01.707264 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:01.711170 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.712578 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.712586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.712590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.712794 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.713443 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.713450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.713453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.713914 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:01.714096 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:01.715314 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0805 22:47:01.715598 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":266,"threads_started":1}
I20260805 22:47:01.717718 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.688969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40619 (local address 127.4.28.3:58315)
0805 22:47:01.689968 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:01.690032 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:01.692172 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:01.693466 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:01.693480 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:01.694398 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:01.695351 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.695380 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.697073 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.697083 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.699603 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.699631 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.700918 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.700931 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.701291 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.702228 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:01.702299 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:01.706721 (+ 4422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:01.711437 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:01.711453 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:01.711487 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:01.712381 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:01.712940 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:01.712946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:01.712949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:01.713279 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:01.714089 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:01.714099 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:01.714734 (+ 635us) client_negotiation.cc:769] Sending connection context
0805 22:47:01.715155 (+ 421us) client_negotiation.cc:240] Negotiation successful
0805 22:47:01.715877 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":440,"threads_started":1}
I20260805 22:47:01.727434 7166 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:38191
I20260805 22:47:01.727524 7363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:38191 every 8 connection(s)
I20260805 22:47:01.729226 7166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:01.731065 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 7166
I20260805 22:47:01.731287 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:47:01.731858 4208 external_mini_cluster.cc:1428] Setting key ddec6ea70bb9a94e695caccb6ccd892a
I20260805 22:47:01.731915 7364 heartbeater.cc:347] Connected to a master server at 127.4.28.62:40619
I20260805 22:47:01.732177 7364 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:01.733042 7364 heartbeater.cc:511] Master 127.4.28.62:40619 requested a full tablet report, sending...
I20260805 22:47:01.734836 6674 ts_manager.cc:194] Registered new tserver with Master: a69d11b65474431c82f000b800174059 (127.4.28.3:38191)
I20260805 22:47:01.735966 6674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:58315
I20260805 22:47:01.740561 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.726044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:56019 (local address 127.4.28.62:40619)
0805 22:47:01.726282 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:01.726289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:01.727083 (+ 794us) server_negotiation.cc:407] Connection header received
0805 22:47:01.727317 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:01.727324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:01.727425 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:01.727560 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:01.728916 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.729763 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.731820 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.732469 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.732556 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.734980 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:01.734987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:01.734991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:01.735042 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:01.737940 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.738637 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.738643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.738648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.738782 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.739139 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.739144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.739147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.739433 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:01.739583 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:01.739972 (+ 389us) server_negotiation.cc:299] Negotiation successful
0805 22:47:01.740198 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260805 22:47:01.740630 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.726055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40619 (local address 127.4.28.3:56019)
0805 22:47:01.726922 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:01.726946 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:01.727148 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:01.727683 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:01.727688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:01.728312 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:01.728728 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.728745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.729938 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.729944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.731624 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.731643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.732850 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.732857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.733027 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.733757 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:01.733786 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:01.734749 (+ 963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:01.738078 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:01.738084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:01.738088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:01.738517 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:01.738892 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:01.738898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:01.738902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:01.739033 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:01.739552 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:01.739557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:01.739729 (+ 172us) client_negotiation.cc:769] Sending connection context
0805 22:47:01.739853 (+ 124us) client_negotiation.cc:240] Negotiation successful
0805 22:47:01.740182 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":496,"threads_started":1}
I20260805 22:47:01.748311 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 22:47:01 dist-test-slave-7lbf krb5kdc[6610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 22:47:01 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:01.805622 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.790994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.4.28.62:40619)
0805 22:47:01.791244 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:01.791250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:01.791433 (+ 183us) server_negotiation.cc:407] Connection header received
0805 22:47:01.791647 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:01.791652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:01.791722 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:01.791815 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:01.793296 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.794087 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.795455 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.795873 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.796001 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.800191 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:01.800197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:01.800200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:01.800261 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:01.802806 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.803556 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.803560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.803562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.803703 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:01.804102 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:01.804106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:01.804108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:01.804398 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:01.804673 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:01.804983 (+ 310us) server_negotiation.cc:299] Negotiation successful
0805 22:47:01.805232 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260805 22:47:01.810155 6674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:01.817221 6674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:01.845721 7392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.833072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:34495 (local address 127.0.0.1:37360)
0805 22:47:01.837017 (+ 3945us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:01.837039 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:01.837193 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:01.837889 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:01.837894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:01.837926 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:01.838441 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.838456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.840160 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.840166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.842252 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.842275 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.844434 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.844441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.844720 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.844784 (+ 64us) client_negotiation.cc:769] Sending connection context
0805 22:47:01.844936 (+ 152us) client_negotiation.cc:240] Negotiation successful
0805 22:47:01.845164 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3807,"mutex_wait_us":61,"thread_start_us":3662,"threads_started":1}
I20260805 22:47:01.845855 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.831320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.4.28.1:37413)
0805 22:47:01.836920 (+ 5600us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:01.836927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:01.836955 (+ 28us) server_negotiation.cc:407] Connection header received
0805 22:47:01.837055 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:01.837061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:01.837341 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:01.838199 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:01.839289 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.840904 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.843418 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.844743 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.844873 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.845104 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:01.845353 (+ 249us) server_negotiation.cc:299] Negotiation successful
0805 22:47:01.845557 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5453,"thread_start_us":5309,"threads_started":1}
I20260805 22:47:01.846547 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.832250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.4.28.3:38191)
0805 22:47:01.837867 (+ 5617us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:01.837876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:01.837906 (+ 30us) server_negotiation.cc:407] Connection header received
0805 22:47:01.838021 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:01.838029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:01.838273 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:01.838475 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:01.839121 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.840904 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.843379 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.844814 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.844945 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.845185 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:01.846037 (+ 852us) server_negotiation.cc:299] Negotiation successful
0805 22:47:01.846216 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5442,"thread_start_us":5255,"threads_started":1}
I20260805 22:47:01.846670 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.833704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.4.28.2:34495)
0805 22:47:01.836555 (+ 2851us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:01.836566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:01.837171 (+ 605us) server_negotiation.cc:407] Connection header received
0805 22:47:01.837351 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:01.837358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:01.837637 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:01.837871 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:01.838654 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.840016 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.842466 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.843934 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.844062 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.844291 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:01.846199 (+ 1908us) server_negotiation.cc:299] Negotiation successful
0805 22:47:01.846388 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2686,"thread_start_us":2563,"threads_started":1}
I20260805 22:47:01.845921 7388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.831043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:37413 (local address 127.0.0.1:41030)
0805 22:47:01.833574 (+ 2531us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:01.833644 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:01.833869 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:01.837672 (+ 3803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:01.837679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:01.837713 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:01.839071 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.839082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.841078 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.841084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.843210 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.843226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.844968 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.844975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.845161 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.845190 (+ 29us) client_negotiation.cc:769] Sending connection context
0805 22:47:01.845280 (+ 90us) client_negotiation.cc:240] Negotiation successful
0805 22:47:01.845430 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2152,"mutex_wait_us":69,"thread_start_us":4874,"threads_started":1}
I20260805 22:47:01.847088 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:01.831418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:38191 (local address 127.0.0.1:42270)
0805 22:47:01.836269 (+ 4851us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:01.836291 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:01.836436 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:01.838462 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:01.838466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:01.838505 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:01.838912 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.838924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.841127 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.841133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.843164 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:01.843182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:01.845342 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:01.845348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:01.845548 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:01.845580 (+ 32us) client_negotiation.cc:769] Sending connection context
0805 22:47:01.845680 (+ 100us) client_negotiation.cc:240] Negotiation successful
0805 22:47:01.845817 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4677,"mutex_wait_us":36,"thread_start_us":3422,"threads_started":1}
I20260805 22:47:01.851395 7058 tablet_service.cc:1467] Processing CreateTablet for tablet 6c194e907c65446391c11cbf782d8b21 (DEFAULT_TABLE table=test-table [id=9637164a2ffd4ef4bb88bc1f653c6805]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:01.851395 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 6c194e907c65446391c11cbf782d8b21 (DEFAULT_TABLE table=test-table [id=9637164a2ffd4ef4bb88bc1f653c6805]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:01.853134 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c194e907c65446391c11cbf782d8b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:01.853163 7058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c194e907c65446391c11cbf782d8b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:01.855417 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 6c194e907c65446391c11cbf782d8b21 (DEFAULT_TABLE table=test-table [id=9637164a2ffd4ef4bb88bc1f653c6805]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:01.856810 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c194e907c65446391c11cbf782d8b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:01.870350 7397 tablet_bootstrap.cc:492] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: Bootstrap starting.
I20260805 22:47:01.870658 7396 tablet_bootstrap.cc:492] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: Bootstrap starting.
I20260805 22:47:01.874246 7397 tablet_bootstrap.cc:654] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:01.874329 7401 tablet_bootstrap.cc:492] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: Bootstrap starting.
I20260805 22:47:01.874632 7396 tablet_bootstrap.cc:654] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:01.875816 7396 log.cc:824] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:01.875844 7397 log.cc:824] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:01.878043 7401 tablet_bootstrap.cc:654] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:01.879221 7397 tablet_bootstrap.cc:492] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: No bootstrap required, opened a new log
I20260805 22:47:01.879388 7396 tablet_bootstrap.cc:492] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: No bootstrap required, opened a new log
I20260805 22:47:01.879412 7401 log.cc:824] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:01.879539 7397 ts_tablet_manager.cc:1397] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260805 22:47:01.879685 7396 ts_tablet_manager.cc:1397] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260805 22:47:01.882964 7401 tablet_bootstrap.cc:492] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: No bootstrap required, opened a new log
I20260805 22:47:01.883313 7401 ts_tablet_manager.cc:1397] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260805 22:47:01.888834 7397 raft_consensus.cc:348] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } }
I20260805 22:47:01.889392 7397 raft_consensus.cc:374] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:01.889580 7397 raft_consensus.cc:729] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d11b65474431c82f000b800174059, State: Initialized, Role: FOLLOWER
I20260805 22:47:01.890277 7397 consensus_queue.cc:260] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } }
I20260805 22:47:01.890436 7396 raft_consensus.cc:348] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } }
I20260805 22:47:01.891017 7396 raft_consensus.cc:374] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:01.891175 7396 raft_consensus.cc:729] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f3ba3f6ee74de98c917371d059be80, State: Initialized, Role: FOLLOWER
I20260805 22:47:01.891559 7364 heartbeater.cc:503] Master 127.4.28.62:40619 was elected leader, sending a full tablet report...
I20260805 22:47:01.892153 7397 ts_tablet_manager.cc:1428] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260805 22:47:01.893095 7396 consensus_queue.cc:260] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } }
I20260805 22:47:01.894510 7159 heartbeater.cc:503] Master 127.4.28.62:40619 was elected leader, sending a full tablet report...
I20260805 22:47:01.894255 7401 raft_consensus.cc:348] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } }
I20260805 22:47:01.894759 7401 raft_consensus.cc:374] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:01.894861 7401 raft_consensus.cc:729] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58d6a5391754278b2c2dfa2537d9e2c, State: Initialized, Role: FOLLOWER
I20260805 22:47:01.895231 7396 ts_tablet_manager.cc:1428] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260805 22:47:01.895637 7401 consensus_queue.cc:260] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } }
I20260805 22:47:01.897661 7401 ts_tablet_manager.cc:1428] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
W20260805 22:47:01.930081 7162 tablet.cc:2369] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:47:01.981287 7365 tablet.cc:2369] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:47:02.061004 7407 raft_consensus.cc:482] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:02.061372 7407 raft_consensus.cc:504] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } }
I20260805 22:47:02.063205 7407 leader_election.cc:288] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413), 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495)
I20260805 22:47:02.069495 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.063510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:34495 (local address 127.4.28.3:54617)
0805 22:47:02.063844 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:02.063867 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:02.064066 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:02.064515 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:02.064519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:02.064565 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:02.064785 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.064798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.065963 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.065967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.067532 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.067551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.068693 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.068699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.068893 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.068920 (+ 27us) client_negotiation.cc:769] Sending connection context
0805 22:47:02.069012 (+ 92us) client_negotiation.cc:240] Negotiation successful
0805 22:47:02.069181 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":28}
I20260805 22:47:02.069483 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.063728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:54617 (local address 127.4.28.2:34495)
0805 22:47:02.063938 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:02.063944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:02.063970 (+ 26us) server_negotiation.cc:407] Connection header received
0805 22:47:02.064257 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:02.064262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:02.064346 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:02.064463 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:02.064966 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.065801 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.067713 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.068496 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.068643 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.068761 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:02.069057 (+ 296us) server_negotiation.cc:299] Negotiation successful
0805 22:47:02.069196 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:47:02.069867 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.063735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:58051 (local address 127.4.28.1:37413)
0805 22:47:02.063938 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:02.063944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:02.064583 (+ 639us) server_negotiation.cc:407] Connection header received
0805 22:47:02.064852 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:02.064858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:02.064934 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:02.065008 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:02.065604 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.066320 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.068099 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.068897 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.069021 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.069130 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:02.069435 (+ 305us) server_negotiation.cc:299] Negotiation successful
0805 22:47:02.069612 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:47:02.069970 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.063608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:37413 (local address 127.4.28.3:58051)
0805 22:47:02.064466 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:02.064493 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:02.064692 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:02.065052 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:02.065056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:02.065082 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:02.065400 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.065410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.066466 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.066471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.067928 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.067941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.069140 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.069147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.069274 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.069299 (+ 25us) client_negotiation.cc:769] Sending connection context
0805 22:47:02.069390 (+ 91us) client_negotiation.cc:240] Negotiation successful
0805 22:47:02.069555 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":1,"thread_start_us":499,"threads_started":1}
I20260805 22:47:02.070853 7082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c194e907c65446391c11cbf782d8b21" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f3ba3f6ee74de98c917371d059be80" is_pre_election: true
I20260805 22:47:02.070973 6885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c194e907c65446391c11cbf782d8b21" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" is_pre_election: true
I20260805 22:47:02.071420 7082 raft_consensus.cc:2454] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 0.
I20260805 22:47:02.071448 6885 raft_consensus.cc:2454] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 0.
I20260805 22:47:02.072139 7189 leader_election.cc:302] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69d11b65474431c82f000b800174059, d58d6a5391754278b2c2dfa2537d9e2c; no voters:
I20260805 22:47:02.072552 7407 raft_consensus.cc:2781] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:02.072721 7407 raft_consensus.cc:482] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:02.072839 7407 raft_consensus.cc:3037] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:02.075188 7407 raft_consensus.cc:504] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } }
I20260805 22:47:02.075852 7407 leader_election.cc:288] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 election: Requested vote from peers d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413), 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495)
I20260805 22:47:02.076543 7082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c194e907c65446391c11cbf782d8b21" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f3ba3f6ee74de98c917371d059be80"
I20260805 22:47:02.076543 6885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c194e907c65446391c11cbf782d8b21" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58d6a5391754278b2c2dfa2537d9e2c"
I20260805 22:47:02.076795 6885 raft_consensus.cc:3037] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:02.076795 7082 raft_consensus.cc:3037] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:02.080271 7082 raft_consensus.cc:2454] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 1.
I20260805 22:47:02.080271 6885 raft_consensus.cc:2454] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 1.
I20260805 22:47:02.080891 7192 leader_election.cc:302] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52f3ba3f6ee74de98c917371d059be80, a69d11b65474431c82f000b800174059, d58d6a5391754278b2c2dfa2537d9e2c; no voters:
I20260805 22:47:02.081281 7407 raft_consensus.cc:2781] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:02.081840 7407 raft_consensus.cc:686] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 1 LEADER]: Becoming Leader. State: Replica: a69d11b65474431c82f000b800174059, State: Running, Role: LEADER
I20260805 22:47:02.082388 7407 consensus_queue.cc:237] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } }
I20260805 22:47:02.087414 6673 catalog_manager.cc:5495] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 reported cstate change: term changed from 0 to 1, leader changed from <none> to a69d11b65474431c82f000b800174059 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "a69d11b65474431c82f000b800174059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } health_report { overall_health: UNKNOWN } } }
W20260805 22:47:02.114219 6956 tablet.cc:2369] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 05 22:47:02 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970019, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:02.186439 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.172307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40619 (local address 127.0.0.1:49604)
0805 22:47:02.173041 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:02.173058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:02.173199 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:02.173485 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:02.173488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:02.174212 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:02.174482 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.174507 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.175535 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.175539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.176600 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.176615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.177441 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.177447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.177619 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.178270 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:02.178289 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:02.180526 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:02.183676 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:02.183694 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:02.183703 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:02.184130 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:02.184486 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:02.184492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:02.184505 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:02.184667 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:02.185386 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:02.185394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:02.185624 (+ 230us) client_negotiation.cc:769] Sending connection context
0805 22:47:02.185782 (+ 158us) client_negotiation.cc:240] Negotiation successful
0805 22:47:02.186109 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":422,"threads_started":1}
I20260805 22:47:02.186985 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.172310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.4.28.62:40619)
0805 22:47:02.172525 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:02.172533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:02.173150 (+ 617us) server_negotiation.cc:407] Connection header received
0805 22:47:02.173297 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:02.173301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:02.173366 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:02.173453 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:02.174641 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.175403 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.176758 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.177250 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.177366 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.180719 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:02.180724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:02.180728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:02.180759 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:02.183516 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:02.184254 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:02.184259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:02.184261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:02.184382 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:02.184836 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:02.184843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:02.184847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:02.185234 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:02.185405 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:02.185915 (+ 510us) server_negotiation.cc:299] Negotiation successful
0805 22:47:02.186333 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":72}
I20260805 22:47:02.192420 6673 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:49604:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260805 22:47:02.193614 6673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:02.201140 6861 tablet_service.cc:1467] Processing CreateTablet for tablet cdff02872aeb42328b7f8f6119a4e66f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c55371184e61465f9b780b114c6a09f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 22:47:02.201498 7262 tablet_service.cc:1467] Processing CreateTablet for tablet cdff02872aeb42328b7f8f6119a4e66f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c55371184e61465f9b780b114c6a09f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 22:47:02.201498 7058 tablet_service.cc:1467] Processing CreateTablet for tablet cdff02872aeb42328b7f8f6119a4e66f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c55371184e61465f9b780b114c6a09f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 22:47:02.202315 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdff02872aeb42328b7f8f6119a4e66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:02.202450 7058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdff02872aeb42328b7f8f6119a4e66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:02.202450 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdff02872aeb42328b7f8f6119a4e66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:02.208957 7401 tablet_bootstrap.cc:492] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: Bootstrap starting.
I20260805 22:47:02.209712 7397 tablet_bootstrap.cc:492] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: Bootstrap starting.
I20260805 22:47:02.209743 7396 tablet_bootstrap.cc:492] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: Bootstrap starting.
I20260805 22:47:02.210891 7401 tablet_bootstrap.cc:654] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:02.211437 7396 tablet_bootstrap.cc:654] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:02.211488 7397 tablet_bootstrap.cc:654] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:02.213580 7401 tablet_bootstrap.cc:492] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: No bootstrap required, opened a new log
I20260805 22:47:02.213737 7397 tablet_bootstrap.cc:492] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: No bootstrap required, opened a new log
I20260805 22:47:02.213763 7401 ts_tablet_manager.cc:1397] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260805 22:47:02.213886 7397 ts_tablet_manager.cc:1397] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260805 22:47:02.213848 7396 tablet_bootstrap.cc:492] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: No bootstrap required, opened a new log
I20260805 22:47:02.214071 7396 ts_tablet_manager.cc:1397] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260805 22:47:02.214617 7397 raft_consensus.cc:348] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.214643 7401 raft_consensus.cc:348] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.214815 7397 raft_consensus.cc:374] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:02.214910 7397 raft_consensus.cc:729] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d11b65474431c82f000b800174059, State: Initialized, Role: FOLLOWER
I20260805 22:47:02.214910 7401 raft_consensus.cc:374] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:02.215000 7401 raft_consensus.cc:729] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58d6a5391754278b2c2dfa2537d9e2c, State: Initialized, Role: FOLLOWER
I20260805 22:47:02.214835 7396 raft_consensus.cc:348] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.215143 7396 raft_consensus.cc:374] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:02.215240 7396 raft_consensus.cc:729] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f3ba3f6ee74de98c917371d059be80, State: Initialized, Role: FOLLOWER
I20260805 22:47:02.215149 7397 consensus_queue.cc:260] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.215250 7401 consensus_queue.cc:260] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.215476 7396 consensus_queue.cc:260] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.215662 7407 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.215864 7401 ts_tablet_manager.cc:1428] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260805 22:47:02.215977 7407 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:02.216068 7396 ts_tablet_manager.cc:1428] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260805 22:47:02.216248 7397 ts_tablet_manager.cc:1428] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260805 22:47:02.216831 7408 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.217274 7408 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:02.217387 7411 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.218024 7411 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:02.435639 7411 raft_consensus.cc:482] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:02.435894 7411 raft_consensus.cc:504] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.437595 7411 leader_election.cc:288] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495), a69d11b65474431c82f000b800174059 (127.4.28.3:38191)
I20260805 22:47:02.447292 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.438094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:58985 (local address 127.4.28.2:34495)
0805 22:47:02.438318 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:02.438327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:02.439100 (+ 773us) server_negotiation.cc:407] Connection header received
0805 22:47:02.439331 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:02.439353 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:02.439438 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:02.439548 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:02.440421 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.441690 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.444788 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.446009 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.446168 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.446280 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:02.446836 (+ 556us) server_negotiation.cc:299] Negotiation successful
0805 22:47:02.446985 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260805 22:47:02.447381 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.438061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:40533 (local address 127.4.28.3:38191)
0805 22:47:02.438698 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:02.438705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:02.439225 (+ 520us) server_negotiation.cc:407] Connection header received
0805 22:47:02.439351 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:02.439356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:02.439437 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:02.439548 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:02.440407 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.441690 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.444678 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.446054 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.446175 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.446289 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:02.446836 (+ 547us) server_negotiation.cc:299] Negotiation successful
0805 22:47:02.447056 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":374,"threads_started":1}
I20260805 22:47:02.447479 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.437902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:34495 (local address 127.4.28.1:58985)
0805 22:47:02.438938 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:02.438961 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:02.439175 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:02.439577 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:02.439582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:02.439643 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:02.440200 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.440215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.441881 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.441885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.444585 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.444598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.446365 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.446371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.446613 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.446689 (+ 76us) client_negotiation.cc:769] Sending connection context
0805 22:47:02.446797 (+ 108us) client_negotiation.cc:240] Negotiation successful
0805 22:47:02.447008 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":458,"spinlock_wait_cycles":31488,"thread_start_us":942,"threads_started":1}
I20260805 22:47:02.447840 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.437988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:38191 (local address 127.4.28.1:40533)
0805 22:47:02.439122 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:02.439137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:02.439240 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:02.439577 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:02.439582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:02.439645 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:02.440161 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.440179 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.441871 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.441877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.444452 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.444474 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.446347 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.446356 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.446607 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.446690 (+ 83us) client_negotiation.cc:769] Sending connection context
0805 22:47:02.446797 (+ 107us) client_negotiation.cc:240] Negotiation successful
0805 22:47:02.447001 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":263,"spinlock_wait_cycles":8576,"thread_start_us":652,"threads_started":1}
I20260805 22:47:02.448423 7082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdff02872aeb42328b7f8f6119a4e66f" candidate_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f3ba3f6ee74de98c917371d059be80" is_pre_election: true
I20260805 22:47:02.448702 7082 raft_consensus.cc:2454] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d58d6a5391754278b2c2dfa2537d9e2c in term 0.
I20260805 22:47:02.448767 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdff02872aeb42328b7f8f6119a4e66f" candidate_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d11b65474431c82f000b800174059" is_pre_election: true
I20260805 22:47:02.449174 7292 raft_consensus.cc:2454] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d58d6a5391754278b2c2dfa2537d9e2c in term 0.
I20260805 22:47:02.449323 6793 leader_election.cc:302] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f3ba3f6ee74de98c917371d059be80, d58d6a5391754278b2c2dfa2537d9e2c; no voters:
I20260805 22:47:02.449805 7411 raft_consensus.cc:2781] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:02.449950 7411 raft_consensus.cc:482] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:02.450052 7411 raft_consensus.cc:3037] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:02.452308 7411 raft_consensus.cc:504] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.452976 7411 leader_election.cc:288] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [CANDIDATE]: Term 1 election: Requested vote from peers 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495), a69d11b65474431c82f000b800174059 (127.4.28.3:38191)
I20260805 22:47:02.453600 7082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdff02872aeb42328b7f8f6119a4e66f" candidate_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f3ba3f6ee74de98c917371d059be80"
I20260805 22:47:02.453708 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdff02872aeb42328b7f8f6119a4e66f" candidate_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d11b65474431c82f000b800174059"
I20260805 22:47:02.453853 7082 raft_consensus.cc:3037] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:02.453933 7292 raft_consensus.cc:3037] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:02.456099 7082 raft_consensus.cc:2454] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d58d6a5391754278b2c2dfa2537d9e2c in term 1.
I20260805 22:47:02.456261 7292 raft_consensus.cc:2454] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d58d6a5391754278b2c2dfa2537d9e2c in term 1.
I20260805 22:47:02.456594 6793 leader_election.cc:302] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f3ba3f6ee74de98c917371d059be80, d58d6a5391754278b2c2dfa2537d9e2c; no voters:
I20260805 22:47:02.456992 7411 raft_consensus.cc:2781] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:02.457474 7411 raft_consensus.cc:686] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [term 1 LEADER]: Becoming Leader. State: Replica: d58d6a5391754278b2c2dfa2537d9e2c, State: Running, Role: LEADER
I20260805 22:47:02.458031 7411 consensus_queue.cc:237] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.461606 7434 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: TxnStatusTablet state changed. Reason: New leader d58d6a5391754278b2c2dfa2537d9e2c. Latest consensus state: current_term: 1 leader_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.462064 7434 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: This TxnStatusTablet replica's current role is: LEADER
I20260805 22:47:02.462829 7436 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260805 22:47:02.463137 7436 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260805 22:47:02.463171 6672 catalog_manager.cc:5495] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c reported cstate change: term changed from 0 to 1, leader changed from <none> to d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1). New cstate: current_term: 1 leader_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } health_report { overall_health: UNKNOWN } } }
I20260805 22:47:02.518079 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.511153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.4.28.1:37413)
0805 22:47:02.511347 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:02.511353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:02.511482 (+ 129us) server_negotiation.cc:407] Connection header received
0805 22:47:02.511882 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:02.511886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:02.511986 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:02.512112 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:02.512577 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.513440 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.515471 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.516040 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.516153 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.516738 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:02.517372 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:02.517484 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:02.517688 (+ 204us) server_negotiation.cc:299] Negotiation successful
0805 22:47:02.517776 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 22:47:02.518143 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.511086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:37413 (local address 127.0.0.1:41046)
0805 22:47:02.511370 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:02.511393 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:02.511592 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:02.512175 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:02.512180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:02.512195 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:47:02.512416 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.512429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.513618 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.513624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.515211 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.515230 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.516230 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.516237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.516444 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.516535 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:47:02.517486 (+ 951us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:47:02.517546 (+ 60us) client_negotiation.cc:769] Sending connection context
0805 22:47:02.517640 (+ 94us) client_negotiation.cc:240] Negotiation successful
0805 22:47:02.517820 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260805 22:47:02.527832 7292 raft_consensus.cc:1261] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Refusing update from remote peer d58d6a5391754278b2c2dfa2537d9e2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:02.527943 7082 raft_consensus.cc:1261] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Refusing update from remote peer d58d6a5391754278b2c2dfa2537d9e2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:02.529112 7434 consensus_queue.cc:1035] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:02.529567 7411 consensus_queue.cc:1035] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:02.540477 7407 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: TxnStatusTablet state changed. Reason: New leader d58d6a5391754278b2c2dfa2537d9e2c. Latest consensus state: current_term: 1 leader_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.540910 7407 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:02.543061 7408 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: TxnStatusTablet state changed. Reason: New leader d58d6a5391754278b2c2dfa2537d9e2c. Latest consensus state: current_term: 1 leader_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.543416 7408 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:02.546250 7442 mvcc.cc:205] Tried to move back new op lower bound from 7315333212261998592 to 7315333211992072192. Current Snapshot: MvccSnapshot[applied={T|T < 7315333212261998592}]
I20260805 22:47:02.547030 7411 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.547343 7411 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: This TxnStatusTablet replica's current role is: LEADER
I20260805 22:47:02.559110 7407 consensus_queue.cc:1035] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:02.566363 7447 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.567690 7447 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P d58d6a5391754278b2c2dfa2537d9e2c: This TxnStatusTablet replica's current role is: LEADER
I20260805 22:47:02.568089 7444 mvcc.cc:205] Tried to move back new op lower bound from 7315333212261998592 to 7315333211992072192. Current Snapshot: MvccSnapshot[applied={T|T < 7315333212261998592}]
I20260805 22:47:02.570998 7407 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.571285 7407 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P a69d11b65474431c82f000b800174059: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:02.578293 7408 tablet_replica.cc:439] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } } }
I20260805 22:47:02.578624 7408 tablet_replica.cc:442] T cdff02872aeb42328b7f8f6119a4e66f P 52f3ba3f6ee74de98c917371d059be80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:02.590466 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.578148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.4.28.3:38191)
0805 22:47:02.578353 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:02.578359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:02.582178 (+ 3819us) server_negotiation.cc:407] Connection header received
0805 22:47:02.582585 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:02.582592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:02.582739 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:02.582923 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:02.583948 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.585130 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.587455 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.588150 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.588283 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.588967 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:02.589780 (+ 813us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:02.589944 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:02.590046 (+ 102us) server_negotiation.cc:299] Negotiation successful
0805 22:47:02.590146 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 22:47:02.596424 7461 consensus_queue.cc:1035] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:02.631565 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.617913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:38861 (local address 127.4.28.1:37413)
0805 22:47:02.618361 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:02.618367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:02.626357 (+ 7990us) server_negotiation.cc:407] Connection header received
0805 22:47:02.626584 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:02.626588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:02.626661 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:02.626751 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:02.627355 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.628176 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.629698 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.630179 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.630298 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.630807 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:02.630921 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:02.631004 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:02.631152 (+ 148us) server_negotiation.cc:299] Negotiation successful
0805 22:47:02.631239 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260805 22:47:02.631647 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.617403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:37413 (local address 127.4.28.3:38861)
0805 22:47:02.626203 (+ 8800us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:02.626227 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:02.626411 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:02.626806 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:02.626812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:02.626835 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:47:02.627180 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.627196 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.628333 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.628340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.629496 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.629557 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.630364 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.630371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.630511 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.630597 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:47:02.631022 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:47:02.631034 (+ 12us) client_negotiation.cc:769] Sending connection context
0805 22:47:02.631114 (+ 80us) client_negotiation.cc:240] Negotiation successful
0805 22:47:02.631268 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8579,"thread_start_us":8349,"threads_started":1}
I20260805 22:47:02.697944 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.672051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:57701 (local address 127.4.28.3:38191)
0805 22:47:02.673404 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:02.673411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:02.690380 (+ 16969us) server_negotiation.cc:407] Connection header received
0805 22:47:02.690584 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:02.690593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:02.690706 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:02.690847 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:02.691525 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.692749 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.695187 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.695983 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.696103 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.696935 (+ 832us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:02.697146 (+ 211us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:02.697216 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:02.697499 (+ 283us) server_negotiation.cc:299] Negotiation successful
0805 22:47:02.697615 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1190}
I20260805 22:47:02.699112 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:02.672078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:38191 (local address 127.4.28.1:57701)
0805 22:47:02.690213 (+ 18135us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:02.690244 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:02.690415 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:02.690864 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:02.690869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:02.690884 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:47:02.691305 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.691320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.692949 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.692956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.694966 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:02.694989 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:02.696365 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:02.696373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:02.696582 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:02.696680 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:47:02.697317 (+ 637us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:47:02.697328 (+ 11us) client_negotiation.cc:769] Sending connection context
0805 22:47:02.698459 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0805 22:47:02.698654 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17859,"thread_start_us":17677,"threads_started":1}
I20260805 22:47:02.746511 6672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:02.746785 6672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:02.751446 6672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f: Sending DeleteTablet for 3 replicas of tablet 6c194e907c65446391c11cbf782d8b21
I20260805 22:47:02.752844 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c194e907c65446391c11cbf782d8b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:02 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260805 22:47:02.753162 7262 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c194e907c65446391c11cbf782d8b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:02 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:42270
I20260805 22:47:02.753207 7058 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c194e907c65446391c11cbf782d8b21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:02 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:37360
I20260805 22:47:02.755679 6672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:02.756472 6672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:02.759301 7477 tablet_replica.cc:330] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: stopping tablet replica
I20260805 22:47:02.760249 7475 tablet_replica.cc:330] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: stopping tablet replica
I20260805 22:47:02.760792 7475 raft_consensus.cc:2229] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:02.761242 7475 raft_consensus.cc:2258] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:02.762988 7478 tablet_replica.cc:330] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: stopping tablet replica
I20260805 22:47:02.763497 7478 raft_consensus.cc:2229] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:02.760241 7477 raft_consensus.cc:2229] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:02.764446 7477 raft_consensus.cc:2258] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:02.765159 7475 ts_tablet_manager.cc:1905] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:02.765296 7478 raft_consensus.cc:2258] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:02.767946 7058 tablet_service.cc:1467] Processing CreateTablet for tablet 98ab673742004915b9372ead5ba53e11 (DEFAULT_TABLE table=test-table [id=dcc54f941f6140ddbc3ca6714454b0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:02.769111 7058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ab673742004915b9372ead5ba53e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:02.774887 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 98ab673742004915b9372ead5ba53e11 (DEFAULT_TABLE table=test-table [id=dcc54f941f6140ddbc3ca6714454b0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:02.775575 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 98ab673742004915b9372ead5ba53e11 (DEFAULT_TABLE table=test-table [id=dcc54f941f6140ddbc3ca6714454b0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:02.776249 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ab673742004915b9372ead5ba53e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:02.779003 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ab673742004915b9372ead5ba53e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:02.779024 7477 ts_tablet_manager.cc:1905] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:02.779160 7479 tablet_bootstrap.cc:492] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: Bootstrap starting.
I20260805 22:47:02.780822 7478 ts_tablet_manager.cc:1905] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:02.783192 7479 tablet_bootstrap.cc:654] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:02.784431 7475 ts_tablet_manager.cc:1918] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:02.784729 7475 log.cc:1196] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/6c194e907c65446391c11cbf782d8b21
I20260805 22:47:02.785511 7475 ts_tablet_manager.cc:1939] T 6c194e907c65446391c11cbf782d8b21 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting consensus metadata
I20260805 22:47:02.787417 7477 ts_tablet_manager.cc:1918] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:02.787647 7477 log.cc:1196] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/6c194e907c65446391c11cbf782d8b21
I20260805 22:47:02.788296 7477 ts_tablet_manager.cc:1939] T 6c194e907c65446391c11cbf782d8b21 P 52f3ba3f6ee74de98c917371d059be80: Deleting consensus metadata
I20260805 22:47:02.790762 6650 catalog_manager.cc:4841] TS 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495): tablet 6c194e907c65446391c11cbf782d8b21 (table test-table [id=9637164a2ffd4ef4bb88bc1f653c6805]) successfully deleted
I20260805 22:47:02.799759 6647 catalog_manager.cc:4841] TS d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413): tablet 6c194e907c65446391c11cbf782d8b21 (table test-table [id=9637164a2ffd4ef4bb88bc1f653c6805]) successfully deleted
I20260805 22:47:02.804028 7479 tablet_bootstrap.cc:492] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: No bootstrap required, opened a new log
I20260805 22:47:02.804278 7479 ts_tablet_manager.cc:1397] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.014s
I20260805 22:47:02.805156 7479 raft_consensus.cc:348] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.805423 7479 raft_consensus.cc:374] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:02.806604 7479 raft_consensus.cc:729] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f3ba3f6ee74de98c917371d059be80, State: Initialized, Role: FOLLOWER
I20260805 22:47:02.806690 7482 tablet_bootstrap.cc:492] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: Bootstrap starting.
I20260805 22:47:02.807112 7479 consensus_queue.cc:260] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.807915 7479 ts_tablet_manager.cc:1428] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260805 22:47:02.814018 7482 tablet_bootstrap.cc:654] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:02.822134 7486 tablet_bootstrap.cc:492] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: Bootstrap starting.
I20260805 22:47:02.822728 7478 ts_tablet_manager.cc:1918] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:02.822963 7478 log.cc:1196] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/6c194e907c65446391c11cbf782d8b21
I20260805 22:47:02.823665 7478 ts_tablet_manager.cc:1939] T 6c194e907c65446391c11cbf782d8b21 P a69d11b65474431c82f000b800174059: Deleting consensus metadata
I20260805 22:47:02.825032 7486 tablet_bootstrap.cc:654] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:02.825865 6651 catalog_manager.cc:4841] TS a69d11b65474431c82f000b800174059 (127.4.28.3:38191): tablet 6c194e907c65446391c11cbf782d8b21 (table test-table [id=9637164a2ffd4ef4bb88bc1f653c6805]) successfully deleted
I20260805 22:47:02.827957 7482 tablet_bootstrap.cc:492] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: No bootstrap required, opened a new log
I20260805 22:47:02.829705 7482 ts_tablet_manager.cc:1397] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.006s
I20260805 22:47:02.830724 7482 raft_consensus.cc:348] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.831197 7482 raft_consensus.cc:374] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:02.831364 7482 raft_consensus.cc:729] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58d6a5391754278b2c2dfa2537d9e2c, State: Initialized, Role: FOLLOWER
I20260805 22:47:02.831785 7482 consensus_queue.cc:260] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.835580 7482 ts_tablet_manager.cc:1428] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260805 22:47:02.841502 7486 tablet_bootstrap.cc:492] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: No bootstrap required, opened a new log
I20260805 22:47:02.841828 7486 ts_tablet_manager.cc:1397] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20260805 22:47:02.842916 7486 raft_consensus.cc:348] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.843281 7486 raft_consensus.cc:374] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:02.843454 7486 raft_consensus.cc:729] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d11b65474431c82f000b800174059, State: Initialized, Role: FOLLOWER
I20260805 22:47:02.843865 7486 consensus_queue.cc:260] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.847587 7486 ts_tablet_manager.cc:1428] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260805 22:47:02.899513 7407 raft_consensus.cc:482] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:02.899785 7407 raft_consensus.cc:504] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.900703 7407 leader_election.cc:288] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413), 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495)
I20260805 22:47:02.901703 6885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ab673742004915b9372ead5ba53e11" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" is_pre_election: true
I20260805 22:47:02.902024 6885 raft_consensus.cc:2454] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 0.
I20260805 22:47:02.902938 7189 leader_election.cc:302] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69d11b65474431c82f000b800174059, d58d6a5391754278b2c2dfa2537d9e2c; no voters:
I20260805 22:47:02.904085 7081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ab673742004915b9372ead5ba53e11" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f3ba3f6ee74de98c917371d059be80" is_pre_election: true
I20260805 22:47:02.904404 7081 raft_consensus.cc:2454] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 0.
I20260805 22:47:02.905929 7407 raft_consensus.cc:2781] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:02.906070 7407 raft_consensus.cc:482] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:02.906180 7407 raft_consensus.cc:3037] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:02.910226 7407 raft_consensus.cc:504] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.911662 6885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ab673742004915b9372ead5ba53e11" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58d6a5391754278b2c2dfa2537d9e2c"
I20260805 22:47:02.911957 6885 raft_consensus.cc:3037] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:02.915791 6885 raft_consensus.cc:2454] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 1.
I20260805 22:47:02.916580 7189 leader_election.cc:302] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69d11b65474431c82f000b800174059, d58d6a5391754278b2c2dfa2537d9e2c; no voters:
I20260805 22:47:02.917040 7462 raft_consensus.cc:2781] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:02.917196 7462 raft_consensus.cc:686] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 1 LEADER]: Becoming Leader. State: Replica: a69d11b65474431c82f000b800174059, State: Running, Role: LEADER
I20260805 22:47:02.917680 7462 consensus_queue.cc:237] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:02.918694 7081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ab673742004915b9372ead5ba53e11" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f3ba3f6ee74de98c917371d059be80"
I20260805 22:47:02.918970 7081 raft_consensus.cc:3037] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:02.922643 7081 raft_consensus.cc:2454] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 1.
I20260805 22:47:02.923473 7407 leader_election.cc:288] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 election: Requested vote from peers d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413), 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495)
I20260805 22:47:02.927749 6672 catalog_manager.cc:5495] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 reported cstate change: term changed from 0 to 1, leader changed from <none> to a69d11b65474431c82f000b800174059 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "a69d11b65474431c82f000b800174059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } health_report { overall_health: HEALTHY } } }
I20260805 22:47:02.974921 7082 raft_consensus.cc:1261] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Refusing update from remote peer a69d11b65474431c82f000b800174059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:02.975849 7407 consensus_queue.cc:1035] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:02.977398 6885 raft_consensus.cc:1261] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Refusing update from remote peer a69d11b65474431c82f000b800174059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:02.978703 7407 consensus_queue.cc:1035] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 22:47:03.030652 7444 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260805 22:47:04.683825 6672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:04.684093 6672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:04.688810 6672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f: Sending DeleteTablet for 3 replicas of tablet 98ab673742004915b9372ead5ba53e11
I20260805 22:47:04.689816 7058 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ab673742004915b9372ead5ba53e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:04 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:37360
I20260805 22:47:04.689817 7262 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ab673742004915b9372ead5ba53e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:04 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:42270
I20260805 22:47:04.689817 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ab673742004915b9372ead5ba53e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:04 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260805 22:47:04.704478 7543 tablet_replica.cc:330] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: stopping tablet replica
I20260805 22:47:04.704649 7536 tablet_replica.cc:330] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: stopping tablet replica
I20260805 22:47:04.704845 7543 raft_consensus.cc:2229] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:04.705179 7543 raft_consensus.cc:2258] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:04.705243 7537 tablet_replica.cc:330] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: stopping tablet replica
I20260805 22:47:04.705440 7537 raft_consensus.cc:2229] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:04.705646 7536 raft_consensus.cc:2229] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:04.705808 7537 raft_consensus.cc:2258] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:04.706101 7536 raft_consensus.cc:2258] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:04.706413 7543 ts_tablet_manager.cc:1905] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:04.707086 7536 ts_tablet_manager.cc:1905] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:04.707258 7537 ts_tablet_manager.cc:1905] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:04.713795 7543 ts_tablet_manager.cc:1918] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 22:47:04.713953 7536 ts_tablet_manager.cc:1918] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 22:47:04.714073 7543 log.cc:1196] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/98ab673742004915b9372ead5ba53e11
I20260805 22:47:04.714157 7536 log.cc:1196] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/98ab673742004915b9372ead5ba53e11
I20260805 22:47:04.714174 7537 ts_tablet_manager.cc:1918] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 22:47:04.714376 7537 log.cc:1196] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/98ab673742004915b9372ead5ba53e11
I20260805 22:47:04.714622 7536 ts_tablet_manager.cc:1939] T 98ab673742004915b9372ead5ba53e11 P a69d11b65474431c82f000b800174059: Deleting consensus metadata
I20260805 22:47:04.714715 7543 ts_tablet_manager.cc:1939] T 98ab673742004915b9372ead5ba53e11 P 52f3ba3f6ee74de98c917371d059be80: Deleting consensus metadata
I20260805 22:47:04.714898 7537 ts_tablet_manager.cc:1939] T 98ab673742004915b9372ead5ba53e11 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting consensus metadata
I20260805 22:47:04.715996 6650 catalog_manager.cc:4841] TS 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495): tablet 98ab673742004915b9372ead5ba53e11 (table test-table [id=dcc54f941f6140ddbc3ca6714454b0f0]) successfully deleted
I20260805 22:47:04.715997 6651 catalog_manager.cc:4841] TS a69d11b65474431c82f000b800174059 (127.4.28.3:38191): tablet 98ab673742004915b9372ead5ba53e11 (table test-table [id=dcc54f941f6140ddbc3ca6714454b0f0]) successfully deleted
I20260805 22:47:04.715996 6647 catalog_manager.cc:4841] TS d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413): tablet 98ab673742004915b9372ead5ba53e11 (table test-table [id=dcc54f941f6140ddbc3ca6714454b0f0]) successfully deleted
Aug 05 22:47:04 dist-test-slave-7lbf krb5kdc[6610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970024, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 05 22:47:04 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970024, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:04.756613 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:04.742298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.4.28.62:40619)
0805 22:47:04.742921 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:04.742925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:04.742951 (+ 26us) server_negotiation.cc:407] Connection header received
0805 22:47:04.743097 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:04.743101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:04.743170 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:04.743267 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:04.744599 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:04.745401 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:04.746707 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:04.747123 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:04.747232 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:04.751497 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:04.751505 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:04.751508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:04.751551 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:04.754258 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:04.754925 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:04.754930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:04.754932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:04.755085 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:04.755388 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:04.755391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:04.755393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:04.755664 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:04.755873 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:04.756123 (+ 250us) server_negotiation.cc:299] Negotiation successful
0805 22:47:04.756299 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":291,"threads_started":1}
I20260805 22:47:04.760354 6672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:04.760977 6672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:04.767923 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 60e6e217e3a8430cb49281eb26014916 (DEFAULT_TABLE table=test-table [id=8d3c8f2dc0fe4e49be0bb3ab746b0e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:04.768085 7058 tablet_service.cc:1467] Processing CreateTablet for tablet 60e6e217e3a8430cb49281eb26014916 (DEFAULT_TABLE table=test-table [id=8d3c8f2dc0fe4e49be0bb3ab746b0e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:04.768154 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 60e6e217e3a8430cb49281eb26014916 (DEFAULT_TABLE table=test-table [id=8d3c8f2dc0fe4e49be0bb3ab746b0e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:04.768487 7058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e6e217e3a8430cb49281eb26014916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:04.768474 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e6e217e3a8430cb49281eb26014916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:04.768657 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e6e217e3a8430cb49281eb26014916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:04.775377 7554 tablet_bootstrap.cc:492] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: Bootstrap starting.
I20260805 22:47:04.776937 7554 tablet_bootstrap.cc:654] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:04.782939 7555 tablet_bootstrap.cc:492] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: Bootstrap starting.
I20260805 22:47:04.783445 7557 tablet_bootstrap.cc:492] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: Bootstrap starting.
I20260805 22:47:04.784991 7557 tablet_bootstrap.cc:654] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:04.785090 7555 tablet_bootstrap.cc:654] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:04.786561 7554 tablet_bootstrap.cc:492] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: No bootstrap required, opened a new log
I20260805 22:47:04.786957 7554 ts_tablet_manager.cc:1397] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260805 22:47:04.788518 7554 raft_consensus.cc:348] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:04.789115 7554 raft_consensus.cc:374] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:04.789323 7554 raft_consensus.cc:729] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d11b65474431c82f000b800174059, State: Initialized, Role: FOLLOWER
I20260805 22:47:04.789867 7554 consensus_queue.cc:260] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:04.792006 7554 ts_tablet_manager.cc:1428] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260805 22:47:04.794878 7555 tablet_bootstrap.cc:492] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: No bootstrap required, opened a new log
I20260805 22:47:04.794876 7557 tablet_bootstrap.cc:492] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: No bootstrap required, opened a new log
I20260805 22:47:04.795063 7557 ts_tablet_manager.cc:1397] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260805 22:47:04.795233 7555 ts_tablet_manager.cc:1397] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260805 22:47:04.795964 7557 raft_consensus.cc:348] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:04.796255 7557 raft_consensus.cc:374] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:04.796356 7557 raft_consensus.cc:729] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f3ba3f6ee74de98c917371d059be80, State: Initialized, Role: FOLLOWER
I20260805 22:47:04.796084 7555 raft_consensus.cc:348] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:04.796579 7555 raft_consensus.cc:374] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:04.796698 7555 raft_consensus.cc:729] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58d6a5391754278b2c2dfa2537d9e2c, State: Initialized, Role: FOLLOWER
I20260805 22:47:04.796651 7557 consensus_queue.cc:260] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:04.796938 7555 consensus_queue.cc:260] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:04.797763 7557 ts_tablet_manager.cc:1428] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260805 22:47:04.797757 7555 ts_tablet_manager.cc:1428] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260805 22:47:04.948427 7527 raft_consensus.cc:482] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:04.948714 7527 raft_consensus.cc:504] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:04.949491 7527 leader_election.cc:288] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413), 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495)
I20260805 22:47:04.950173 6885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e6e217e3a8430cb49281eb26014916" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" is_pre_election: true
I20260805 22:47:04.950188 7081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e6e217e3a8430cb49281eb26014916" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f3ba3f6ee74de98c917371d059be80" is_pre_election: true
I20260805 22:47:04.950450 6885 raft_consensus.cc:2454] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 0.
I20260805 22:47:04.950461 7081 raft_consensus.cc:2454] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 0.
I20260805 22:47:04.950968 7192 leader_election.cc:302] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f3ba3f6ee74de98c917371d059be80, a69d11b65474431c82f000b800174059; no voters:
I20260805 22:47:04.951280 7527 raft_consensus.cc:2781] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:04.951393 7527 raft_consensus.cc:482] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:04.951463 7527 raft_consensus.cc:3037] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:04.953775 7527 raft_consensus.cc:504] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:04.954421 7527 leader_election.cc:288] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 election: Requested vote from peers d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413), 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495)
I20260805 22:47:04.955024 7081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e6e217e3a8430cb49281eb26014916" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f3ba3f6ee74de98c917371d059be80"
I20260805 22:47:04.955029 6885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e6e217e3a8430cb49281eb26014916" candidate_uuid: "a69d11b65474431c82f000b800174059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58d6a5391754278b2c2dfa2537d9e2c"
I20260805 22:47:04.955274 6885 raft_consensus.cc:3037] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:04.955273 7081 raft_consensus.cc:3037] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:04.957484 7081 raft_consensus.cc:2454] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 1.
I20260805 22:47:04.957482 6885 raft_consensus.cc:2454] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69d11b65474431c82f000b800174059 in term 1.
I20260805 22:47:04.958025 7192 leader_election.cc:302] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f3ba3f6ee74de98c917371d059be80, a69d11b65474431c82f000b800174059; no voters:
I20260805 22:47:04.958360 7527 raft_consensus.cc:2781] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:04.958504 7527 raft_consensus.cc:686] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 1 LEADER]: Becoming Leader. State: Replica: a69d11b65474431c82f000b800174059, State: Running, Role: LEADER
I20260805 22:47:04.958796 7527 consensus_queue.cc:237] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:04.961685 6673 catalog_manager.cc:5495] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 reported cstate change: term changed from 0 to 1, leader changed from <none> to a69d11b65474431c82f000b800174059 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "a69d11b65474431c82f000b800174059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } health_report { overall_health: HEALTHY } } }
I20260805 22:47:04.988451 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:04.983505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.4.28.3:38191)
0805 22:47:04.984090 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:04.984094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:04.984115 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:47:04.984173 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:04.984177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:04.984240 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:04.984322 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:04.984661 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:04.985426 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:04.986805 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:04.987281 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:04.987409 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:04.987763 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:04.987870 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:04.987940 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:04.988034 (+ 94us) server_negotiation.cc:299] Negotiation successful
0805 22:47:04.988098 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":339,"threads_started":1}
I20260805 22:47:04.998353 7081 raft_consensus.cc:1261] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Refusing update from remote peer a69d11b65474431c82f000b800174059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:04.998353 6885 raft_consensus.cc:1261] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Refusing update from remote peer a69d11b65474431c82f000b800174059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:04.998925 7515 consensus_queue.cc:1035] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:04.999290 7527 consensus_queue.cc:1035] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:05.047919 6672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:05.048187 6672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:05.051453 6672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f: Sending DeleteTablet for 3 replicas of tablet 60e6e217e3a8430cb49281eb26014916
I20260805 22:47:05.052421 7058 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e6e217e3a8430cb49281eb26014916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:05 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:37360
I20260805 22:47:05.052420 7262 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e6e217e3a8430cb49281eb26014916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:05 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:42270
I20260805 22:47:05.052404 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e6e217e3a8430cb49281eb26014916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:05 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260805 22:47:05.052736 7536 tablet_replica.cc:330] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: stopping tablet replica
I20260805 22:47:05.052736 7543 tablet_replica.cc:330] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: stopping tablet replica
I20260805 22:47:05.052735 7537 tablet_replica.cc:330] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: stopping tablet replica
I20260805 22:47:05.052934 7543 raft_consensus.cc:2229] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:05.052945 7536 raft_consensus.cc:2229] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:05.052938 7537 raft_consensus.cc:2229] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:05.053156 7543 raft_consensus.cc:2258] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:05.053192 7537 raft_consensus.cc:2258] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:05.053334 7536 raft_consensus.cc:2258] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:05.053700 6672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:05.054513 6672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:05.054584 7543 ts_tablet_manager.cc:1905] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:05.054613 7537 ts_tablet_manager.cc:1905] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:05.056303 7536 ts_tablet_manager.cc:1905] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:05.060662 7543 ts_tablet_manager.cc:1918] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:05.060828 7543 log.cc:1196] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/60e6e217e3a8430cb49281eb26014916
I20260805 22:47:05.061256 7537 ts_tablet_manager.cc:1918] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:05.061291 7543 ts_tablet_manager.cc:1939] T 60e6e217e3a8430cb49281eb26014916 P 52f3ba3f6ee74de98c917371d059be80: Deleting consensus metadata
I20260805 22:47:05.061430 7537 log.cc:1196] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/60e6e217e3a8430cb49281eb26014916
I20260805 22:47:05.061880 7537 ts_tablet_manager.cc:1939] T 60e6e217e3a8430cb49281eb26014916 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting consensus metadata
I20260805 22:47:05.062716 6650 catalog_manager.cc:4841] TS 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495): tablet 60e6e217e3a8430cb49281eb26014916 (table test-table [id=8d3c8f2dc0fe4e49be0bb3ab746b0e3b]) successfully deleted
I20260805 22:47:05.062922 6647 catalog_manager.cc:4841] TS d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413): tablet 60e6e217e3a8430cb49281eb26014916 (table test-table [id=8d3c8f2dc0fe4e49be0bb3ab746b0e3b]) successfully deleted
I20260805 22:47:05.063314 7536 ts_tablet_manager.cc:1918] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:05.063479 7536 log.cc:1196] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/60e6e217e3a8430cb49281eb26014916
I20260805 22:47:05.063903 7536 ts_tablet_manager.cc:1939] T 60e6e217e3a8430cb49281eb26014916 P a69d11b65474431c82f000b800174059: Deleting consensus metadata
I20260805 22:47:05.064893 6651 catalog_manager.cc:4841] TS a69d11b65474431c82f000b800174059 (127.4.28.3:38191): tablet 60e6e217e3a8430cb49281eb26014916 (table test-table [id=8d3c8f2dc0fe4e49be0bb3ab746b0e3b]) successfully deleted
I20260805 22:47:05.065987 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 06806a3067f047e7b440cf4a8242bd40 (DEFAULT_TABLE table=test-table [id=4d1d8ef2aafe466b9aadae3662a5310e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:05.066274 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 06806a3067f047e7b440cf4a8242bd40 (DEFAULT_TABLE table=test-table [id=4d1d8ef2aafe466b9aadae3662a5310e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:05.066249 7058 tablet_service.cc:1467] Processing CreateTablet for tablet 06806a3067f047e7b440cf4a8242bd40 (DEFAULT_TABLE table=test-table [id=4d1d8ef2aafe466b9aadae3662a5310e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:05.066381 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06806a3067f047e7b440cf4a8242bd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:05.066694 7058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06806a3067f047e7b440cf4a8242bd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:05.066692 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06806a3067f047e7b440cf4a8242bd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:05.071439 7555 tablet_bootstrap.cc:492] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: Bootstrap starting.
I20260805 22:47:05.072942 7555 tablet_bootstrap.cc:654] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:05.073292 7554 tablet_bootstrap.cc:492] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: Bootstrap starting.
I20260805 22:47:05.073421 7557 tablet_bootstrap.cc:492] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: Bootstrap starting.
I20260805 22:47:05.075047 7554 tablet_bootstrap.cc:654] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:05.075047 7557 tablet_bootstrap.cc:654] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:05.075261 7555 tablet_bootstrap.cc:492] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: No bootstrap required, opened a new log
I20260805 22:47:05.075543 7555 ts_tablet_manager.cc:1397] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260805 22:47:05.076408 7555 raft_consensus.cc:348] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:05.076722 7555 raft_consensus.cc:374] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:05.076814 7555 raft_consensus.cc:729] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58d6a5391754278b2c2dfa2537d9e2c, State: Initialized, Role: FOLLOWER
I20260805 22:47:05.077080 7555 consensus_queue.cc:260] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:05.078250 7555 ts_tablet_manager.cc:1428] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260805 22:47:05.079003 7557 tablet_bootstrap.cc:492] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: No bootstrap required, opened a new log
I20260805 22:47:05.079171 7557 ts_tablet_manager.cc:1397] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260805 22:47:05.079880 7557 raft_consensus.cc:348] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:05.080137 7554 tablet_bootstrap.cc:492] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: No bootstrap required, opened a new log
I20260805 22:47:05.080171 7557 raft_consensus.cc:374] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:05.080272 7557 raft_consensus.cc:729] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f3ba3f6ee74de98c917371d059be80, State: Initialized, Role: FOLLOWER
I20260805 22:47:05.080315 7554 ts_tablet_manager.cc:1397] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260805 22:47:05.080523 7557 consensus_queue.cc:260] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:05.081091 7557 ts_tablet_manager.cc:1428] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260805 22:47:05.081125 7554 raft_consensus.cc:348] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:05.081387 7554 raft_consensus.cc:374] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:05.081489 7554 raft_consensus.cc:729] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d11b65474431c82f000b800174059, State: Initialized, Role: FOLLOWER
I20260805 22:47:05.081830 7554 consensus_queue.cc:260] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:05.082382 7554 ts_tablet_manager.cc:1428] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260805 22:47:05.137415 7564 raft_consensus.cc:482] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:05.137724 7564 raft_consensus.cc:504] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:05.139425 7564 leader_election.cc:288] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413), a69d11b65474431c82f000b800174059 (127.4.28.3:38191)
I20260805 22:47:05.146927 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:05.139772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:37413 (local address 127.4.28.2:34433)
0805 22:47:05.140575 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:05.140597 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:05.140747 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:05.141279 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:05.141284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:05.141310 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:05.141655 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:05.141670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.143045 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.143050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:05.144773 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:05.144787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.146109 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.146116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:05.146293 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:05.146334 (+ 41us) client_negotiation.cc:769] Sending connection context
0805 22:47:05.146438 (+ 104us) client_negotiation.cc:240] Negotiation successful
0805 22:47:05.146563 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":47,"thread_start_us":1223,"threads_started":1}
I20260805 22:47:05.146970 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:05.140273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:34433 (local address 127.4.28.1:37413)
0805 22:47:05.140874 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:05.140880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:05.140908 (+ 28us) server_negotiation.cc:407] Connection header received
0805 22:47:05.140985 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:05.140990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:05.141070 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:05.141214 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:05.141876 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.142892 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.144943 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.145912 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.146023 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:05.146115 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:05.146477 (+ 362us) server_negotiation.cc:299] Negotiation successful
0805 22:47:05.146638 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":275,"threads_started":1}
I20260805 22:47:05.147148 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:05.139849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:59509 (local address 127.4.28.3:38191)
0805 22:47:05.140094 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:05.140101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:05.141467 (+ 1366us) server_negotiation.cc:407] Connection header received
0805 22:47:05.141643 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:05.141648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:05.141751 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:05.141844 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:05.142501 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.143368 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.145255 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.146178 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.146306 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:05.146407 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:05.146765 (+ 358us) server_negotiation.cc:299] Negotiation successful
0805 22:47:05.146903 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260805 22:47:05.147290 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:05.139815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:38191 (local address 127.4.28.2:59509)
0805 22:47:05.141340 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:05.141361 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:05.141481 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:05.141874 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:05.141878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:05.141903 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:05.142351 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:05.142364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.143510 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.143514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:05.145088 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:05.145103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.146388 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.146393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:05.146589 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:05.146623 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 22:47:05.146718 (+ 95us) client_negotiation.cc:240] Negotiation successful
0805 22:47:05.146853 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"mutex_wait_us":22,"thread_start_us":445,"threads_started":1}
I20260805 22:47:05.148033 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06806a3067f047e7b440cf4a8242bd40" candidate_uuid: "52f3ba3f6ee74de98c917371d059be80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d11b65474431c82f000b800174059" is_pre_election: true
I20260805 22:47:05.148029 6885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06806a3067f047e7b440cf4a8242bd40" candidate_uuid: "52f3ba3f6ee74de98c917371d059be80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" is_pre_election: true
I20260805 22:47:05.148306 7292 raft_consensus.cc:2454] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f3ba3f6ee74de98c917371d059be80 in term 0.
I20260805 22:47:05.148331 6885 raft_consensus.cc:2454] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f3ba3f6ee74de98c917371d059be80 in term 0.
I20260805 22:47:05.148921 6986 leader_election.cc:302] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f3ba3f6ee74de98c917371d059be80, d58d6a5391754278b2c2dfa2537d9e2c; no voters:
I20260805 22:47:05.149358 7564 raft_consensus.cc:2781] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:05.149477 7564 raft_consensus.cc:482] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:05.149613 7564 raft_consensus.cc:3037] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:05.151860 7564 raft_consensus.cc:504] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:05.152493 7564 leader_election.cc:288] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [CANDIDATE]: Term 1 election: Requested vote from peers d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413), a69d11b65474431c82f000b800174059 (127.4.28.3:38191)
I20260805 22:47:05.153100 6885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06806a3067f047e7b440cf4a8242bd40" candidate_uuid: "52f3ba3f6ee74de98c917371d059be80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58d6a5391754278b2c2dfa2537d9e2c"
I20260805 22:47:05.153142 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06806a3067f047e7b440cf4a8242bd40" candidate_uuid: "52f3ba3f6ee74de98c917371d059be80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d11b65474431c82f000b800174059"
I20260805 22:47:05.153359 6885 raft_consensus.cc:3037] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:05.153384 7292 raft_consensus.cc:3037] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:05.155655 6885 raft_consensus.cc:2454] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f3ba3f6ee74de98c917371d059be80 in term 1.
I20260805 22:47:05.156194 6986 leader_election.cc:302] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f3ba3f6ee74de98c917371d059be80, d58d6a5391754278b2c2dfa2537d9e2c; no voters:
I20260805 22:47:05.156497 7292 raft_consensus.cc:2454] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f3ba3f6ee74de98c917371d059be80 in term 1.
I20260805 22:47:05.156538 7564 raft_consensus.cc:2781] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:05.157073 7564 raft_consensus.cc:686] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 1 LEADER]: Becoming Leader. State: Replica: 52f3ba3f6ee74de98c917371d059be80, State: Running, Role: LEADER
I20260805 22:47:05.157491 7564 consensus_queue.cc:237] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } }
I20260805 22:47:05.161809 6672 catalog_manager.cc:5495] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2). New cstate: current_term: 1 leader_uuid: "52f3ba3f6ee74de98c917371d059be80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f3ba3f6ee74de98c917371d059be80" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 } health_report { overall_health: UNKNOWN } } }
I20260805 22:47:05.200722 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:05.194360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.4.28.2:34495)
0805 22:47:05.194984 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:05.194989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:05.195009 (+ 20us) server_negotiation.cc:407] Connection header received
0805 22:47:05.195099 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:05.195103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:05.195176 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:05.195335 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:05.195760 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.196672 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.198165 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.198680 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.198798 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:05.199366 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:05.200009 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:05.200157 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:05.200267 (+ 110us) server_negotiation.cc:299] Negotiation successful
0805 22:47:05.200379 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":351,"threads_started":1}
I20260805 22:47:05.218847 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:05.212824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:58199 (local address 127.4.28.1:37413)
0805 22:47:05.213052 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:05.213057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:05.213738 (+ 681us) server_negotiation.cc:407] Connection header received
0805 22:47:05.213956 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:05.213961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:05.214052 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:05.214133 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:05.214687 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.215436 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.216928 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.217412 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.217579 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:05.218087 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:05.218214 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:05.218296 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:05.218442 (+ 146us) server_negotiation.cc:299] Negotiation successful
0805 22:47:05.218532 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:47:05.218946 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:05.212831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:37413 (local address 127.4.28.2:58199)
0805 22:47:05.213617 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:05.213636 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:05.213811 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:05.214170 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:05.214174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:05.214186 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:47:05.214506 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:05.214516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.215601 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.215605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:05.216754 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:05.216769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.217657 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.217663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:05.217801 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:05.217875 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:47:05.218320 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:47:05.218333 (+ 13us) client_negotiation.cc:769] Sending connection context
0805 22:47:05.218407 (+ 74us) client_negotiation.cc:240] Negotiation successful
0805 22:47:05.218584 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":379,"threads_started":1}
I20260805 22:47:05.228228 7292 raft_consensus.cc:1261] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Refusing update from remote peer 52f3ba3f6ee74de98c917371d059be80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:05.228291 6885 raft_consensus.cc:1261] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Refusing update from remote peer 52f3ba3f6ee74de98c917371d059be80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:05.228940 7585 consensus_queue.cc:1035] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69d11b65474431c82f000b800174059" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:05.229310 7564 consensus_queue.cc:1035] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58d6a5391754278b2c2dfa2537d9e2c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:05.234114 7505 mvcc.cc:205] Tried to move back new op lower bound from 7315333223328821248 to 7315333223050272768. Current Snapshot: MvccSnapshot[applied={T|T < 7315333223328821248}]
I20260805 22:47:05.258289 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:05.252025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:39427 (local address 127.4.28.2:34495)
0805 22:47:05.252206 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:05.252214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:05.252878 (+ 664us) server_negotiation.cc:407] Connection header received
0805 22:47:05.253057 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:05.253063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:05.253154 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:05.253240 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:05.253681 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.254666 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.256328 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.256867 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.256994 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:05.257467 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:05.257634 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:05.257716 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:05.257875 (+ 159us) server_negotiation.cc:299] Negotiation successful
0805 22:47:05.257961 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260805 22:47:05.258348 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:05.251954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:34495 (local address 127.4.28.1:39427)
0805 22:47:05.252754 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:05.252774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:05.252931 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:05.253255 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:05.253257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:05.253266 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:47:05.253539 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:05.253551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.254818 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.254824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:05.256135 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:05.256153 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:05.257081 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:05.257088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:05.257235 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:05.257278 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:47:05.257733 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:47:05.257746 (+ 13us) client_negotiation.cc:769] Sending connection context
0805 22:47:05.257821 (+ 75us) client_negotiation.cc:240] Negotiation successful
0805 22:47:05.257980 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":483,"threads_started":1}
W20260805 22:47:05.259696 7505 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260805 22:47:06.881852 6673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:06.882125 6673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:06.887059 6673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 352b2b1af1174c948aba0d7076f3514f: Sending DeleteTablet for 3 replicas of tablet 06806a3067f047e7b440cf4a8242bd40
I20260805 22:47:06.888103 7058 tablet_service.cc:1514] Processing DeleteTablet for tablet 06806a3067f047e7b440cf4a8242bd40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:06 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:37360
I20260805 22:47:06.888155 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 06806a3067f047e7b440cf4a8242bd40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:06 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260805 22:47:06.888159 7262 tablet_service.cc:1514] Processing DeleteTablet for tablet 06806a3067f047e7b440cf4a8242bd40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:06 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:42270
I20260805 22:47:06.888892 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 6771
I20260805 22:47:06.900856 7641 tablet_replica.cc:330] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: stopping tablet replica
I20260805 22:47:06.900916 7639 tablet_replica.cc:330] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: stopping tablet replica
I20260805 22:47:06.901171 7639 raft_consensus.cc:2229] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:06.901446 7641 raft_consensus.cc:2229] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:06.901901 7641 raft_consensus.cc:2258] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:06.901924 7639 raft_consensus.cc:2258] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:06.902647 7640 tablet_replica.cc:330] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: stopping tablet replica
I20260805 22:47:06.902956 7640 raft_consensus.cc:2229] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:06.903302 7640 raft_consensus.cc:2258] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:06.903605 7639 ts_tablet_manager.cc:1905] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:06.904891 7640 ts_tablet_manager.cc:1905] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:06.910912 7639 ts_tablet_manager.cc:1918] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 22:47:06.911098 7639 log.cc:1196] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/06806a3067f047e7b440cf4a8242bd40
I20260805 22:47:06.911602 7639 ts_tablet_manager.cc:1939] T 06806a3067f047e7b440cf4a8242bd40 P 52f3ba3f6ee74de98c917371d059be80: Deleting consensus metadata
I20260805 22:47:06.911994 7640 ts_tablet_manager.cc:1918] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 22:47:06.912150 7640 log.cc:1196] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/06806a3067f047e7b440cf4a8242bd40
I20260805 22:47:06.912642 7640 ts_tablet_manager.cc:1939] T 06806a3067f047e7b440cf4a8242bd40 P a69d11b65474431c82f000b800174059: Deleting consensus metadata
I20260805 22:47:06.913036 6650 catalog_manager.cc:4841] TS 52f3ba3f6ee74de98c917371d059be80 (127.4.28.2:34495): tablet 06806a3067f047e7b440cf4a8242bd40 (table test-table [id=4d1d8ef2aafe466b9aadae3662a5310e]) successfully deleted
I20260805 22:47:06.913177 7641 ts_tablet_manager.cc:1905] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:06.913857 6651 catalog_manager.cc:4841] TS a69d11b65474431c82f000b800174059 (127.4.28.3:38191): tablet 06806a3067f047e7b440cf4a8242bd40 (table test-table [id=4d1d8ef2aafe466b9aadae3662a5310e]) successfully deleted
Aug 05 22:47:06 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970024, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:47:06.919235 7641 ts_tablet_manager.cc:1918] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 22:47:06.919391 7641 log.cc:1196] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/06806a3067f047e7b440cf4a8242bd40
I20260805 22:47:06.919876 7641 ts_tablet_manager.cc:1939] T 06806a3067f047e7b440cf4a8242bd40 P d58d6a5391754278b2c2dfa2537d9e2c: Deleting consensus metadata
I20260805 22:47:06.921130 6647 catalog_manager.cc:4841] TS d58d6a5391754278b2c2dfa2537d9e2c (127.4.28.1:37413): tablet 06806a3067f047e7b440cf4a8242bd40 (table test-table [id=4d1d8ef2aafe466b9aadae3662a5310e]) successfully deleted
I20260805 22:47:06.926329 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:06.891333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.4.28.1:37413)
0805 22:47:06.902863 (+ 11530us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:06.902871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:06.903671 (+ 800us) server_negotiation.cc:407] Connection header received
0805 22:47:06.903746 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:06.903751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:06.903862 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:06.903971 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:06.905874 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:06.907011 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:06.909329 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:06.910016 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:06.910143 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:06.920575 (+ 10432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:06.920599 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:06.920625 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:06.920676 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:06.923778 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:06.924510 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:06.924517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:06.924519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:06.924675 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:06.924970 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:06.924973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:06.924975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:06.925349 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:06.925669 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:06.925869 (+ 200us) server_negotiation.cc:299] Negotiation successful
0805 22:47:06.926056 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11326,"thread_start_us":11064,"threads_started":1}
W20260805 22:47:06.927042 6949 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41048
W20260805 22:47:06.927716 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 6771: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:47:06.927910 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 6771
I20260805 22:47:06.957434 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 6961
Aug 05 22:47:06 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970024, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:47:06.972937 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:06.958279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.4.28.2:34495)
0805 22:47:06.958858 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:06.958863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:06.958880 (+ 17us) server_negotiation.cc:407] Connection header received
0805 22:47:06.958944 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:06.958950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:06.959027 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:06.959132 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:06.960389 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:06.961162 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:06.962457 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:06.962890 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:06.962991 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:06.967269 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:06.967287 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:06.967310 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:06.967355 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:06.970409 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:06.971150 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:06.971157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:06.971160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:06.971297 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:06.971601 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:06.971604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:06.971606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:06.972000 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:06.972241 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:06.972476 (+ 235us) server_negotiation.cc:299] Negotiation successful
0805 22:47:06.972670 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":313,"threads_started":1}
W20260805 22:47:06.973766 7152 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37372
W20260805 22:47:06.974460 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 6961: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:47:06.974654 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 6961
I20260805 22:47:07.003456 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 7166
Aug 05 22:47:07 dist-test-slave-7lbf krb5kdc[6610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970024, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:47:07.019887 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:07.004293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.4.28.3:38191)
0805 22:47:07.004836 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:07.004841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:07.004859 (+ 18us) server_negotiation.cc:407] Connection header received
0805 22:47:07.004919 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:07.004923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:07.004985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:07.005084 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:07.006449 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:07.007258 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:07.008459 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:07.008870 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:07.008971 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:07.013431 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:07.013451 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:07.013471 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:07.013575 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:07.016425 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:07.017302 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:07.017313 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:07.017316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:07.017520 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:07.018091 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:07.018095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:07.018098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:07.018579 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:07.018872 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:07.019230 (+ 358us) server_negotiation.cc:299] Negotiation successful
0805 22:47:07.019449 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":279,"threads_started":1}
W20260805 22:47:07.020606 7354 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42298
W20260805 22:47:07.021313 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 7166: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:47:07.021466 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 7166
I20260805 22:47:07.052651 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 6628
I20260805 22:47:07.054143 6729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:07.171485 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 6628
2026-08-05T22:47:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9164 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:07 dist-test-slave-7lbf krb5kdc[7653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:07 dist-test-slave-7lbf krb5kdc[7653](info): set up 2 sockets
Aug 05 22:47:07 dist-test-slave-7lbf krb5kdc[7653](info): commencing operation
krb5kdc: starting...
W20260805 22:47:08.365869 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:08 dist-test-slave-7lbf krb5kdc[7653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:47:08.605829 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:42895
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:42895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:08.952683 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:08.953033 7669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:08.953168 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:08.960630 7669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:08.960788 7669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:08.960863 7669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:08.960954 7669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:08.961024 7669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:08.971329 7669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:42895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:42895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:08.972805 7669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:08.974298 7669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:08.987068 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:08.987287 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:08.986928 7678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:08.988713 7669 server_base.cc:1034] running on GCE node
I20260805 22:47:08.989776 7669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:08.991291 7669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:08.992547 7669 hybrid_clock.cc:648] HybridClock initialized: now 1785970028992463 us; error 58 us; skew 500 ppm
I20260805 22:47:08.993007 7669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:08 dist-test-slave-7lbf krb5kdc[7653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970028, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:08.996372 7669 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:47:08.997954 7669 webserver.cc:460] Webserver started at http://127.4.28.62:46389/ using document root <none> and password file <none>
I20260805 22:47:08.998647 7669 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:08.998754 7669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:08.999045 7669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:09.002487 7669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "418a4a203256461cb59e748b927346c6"
format_stamp: "Formatted at 2026-08-05 22:47:08 on dist-test-slave-7lbf"
server_key: "19494f7f7ee77b427c31001386cf31c5"
server_key_iv: "8e4d4b3793b6c6535d616cecfec8d7c7"
server_key_version: "encryptionkey@0"
I20260805 22:47:09.003321 7669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "418a4a203256461cb59e748b927346c6"
format_stamp: "Formatted at 2026-08-05 22:47:08 on dist-test-slave-7lbf"
server_key: "19494f7f7ee77b427c31001386cf31c5"
server_key_iv: "8e4d4b3793b6c6535d616cecfec8d7c7"
server_key_version: "encryptionkey@0"
I20260805 22:47:09.009099 7669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260805 22:47:09.013088 7688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:09.014644 7669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:09.014863 7669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "418a4a203256461cb59e748b927346c6"
format_stamp: "Formatted at 2026-08-05 22:47:08 on dist-test-slave-7lbf"
server_key: "19494f7f7ee77b427c31001386cf31c5"
server_key_iv: "8e4d4b3793b6c6535d616cecfec8d7c7"
server_key_version: "encryptionkey@0"
I20260805 22:47:09.015111 7669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:09.042765 7669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:09.047797 7669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:09.048134 7669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:09.067215 7669 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:42895
I20260805 22:47:09.067258 7768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:42895 every 8 connection(s)
I20260805 22:47:09.068722 7669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:09.070770 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 7669
I20260805 22:47:09.071008 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:09.071848 4208 external_mini_cluster.cc:1428] Setting key 3363655554cd5168561b2a39ace51bef
I20260805 22:47:09.074188 7770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:47:09 dist-test-slave-7lbf krb5kdc[7653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:09.088769 7770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6: Bootstrap starting.
I20260805 22:47:09.092722 7770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:09.094197 7770 log.cc:824] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:09.097944 7770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6: No bootstrap required, opened a new log
I20260805 22:47:09.098378 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:09.074211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.4.28.62:42895)
0805 22:47:09.075172 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:09.075189 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:09.075246 (+ 57us) server_negotiation.cc:407] Connection header received
0805 22:47:09.077466 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:09.077501 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:09.078146 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:09.078657 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:09.080162 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:09.082324 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:09.083843 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:09.084905 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:09.085120 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:09.089676 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:09.089687 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:09.089708 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:09.089757 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:09.093160 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:09.094310 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:09.094319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:09.094327 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:09.094558 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:09.095070 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:09.095076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:09.095079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:09.095891 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:09.096132 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:09.096749 (+ 617us) server_negotiation.cc:299] Negotiation successful
0805 22:47:09.097190 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":422,"threads_started":1}
I20260805 22:47:09.106376 7770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418a4a203256461cb59e748b927346c6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42895 } }
I20260805 22:47:09.106904 7770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:09.107023 7770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418a4a203256461cb59e748b927346c6, State: Initialized, Role: FOLLOWER
I20260805 22:47:09.107741 7770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418a4a203256461cb59e748b927346c6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42895 } }
I20260805 22:47:09.107982 7770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:09.108196 7770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:09.108381 7770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:09.111433 7770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418a4a203256461cb59e748b927346c6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42895 } }
I20260805 22:47:09.112102 7770 leader_election.cc:302] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 418a4a203256461cb59e748b927346c6; no voters:
I20260805 22:47:09.112776 7770 leader_election.cc:288] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:09.112892 7780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:09.114156 7780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [term 1 LEADER]: Becoming Leader. State: Replica: 418a4a203256461cb59e748b927346c6, State: Running, Role: LEADER
I20260805 22:47:09.114776 7780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418a4a203256461cb59e748b927346c6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42895 } }
I20260805 22:47:09.115274 7770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:09.118108 7782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 418a4a203256461cb59e748b927346c6. Latest consensus state: current_term: 1 leader_uuid: "418a4a203256461cb59e748b927346c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418a4a203256461cb59e748b927346c6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42895 } } }
I20260805 22:47:09.118614 7782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:09.118194 7781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "418a4a203256461cb59e748b927346c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418a4a203256461cb59e748b927346c6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42895 } } }
I20260805 22:47:09.118747 7781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:09.121395 7793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:09.126767 7793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:09.136691 7793 catalog_manager.cc:1347] Generated new cluster ID: f6990c4956cb4e599165f1cf67dd27f4
I20260805 22:47:09.136830 7793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:09.152679 7793 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:09.154155 7793 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:09.163326 7793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 418a4a203256461cb59e748b927346c6: Generated new TSK 0
I20260805 22:47:09.164281 7793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:47:09.237649 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:42895
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:09.614764 7809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:09.615132 7809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:09.615263 7809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:09.623512 7809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:09.623662 7809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:09.623791 7809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:09.634166 7809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:42895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:09.635583 7809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:09.636969 7809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:09.651069 7818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:09.651031 7817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:09.651105 7820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:09.652172 7809 server_base.cc:1034] running on GCE node
I20260805 22:47:09.652848 7809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:09.654002 7809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:09.655221 7809 hybrid_clock.cc:648] HybridClock initialized: now 1785970029655146 us; error 61 us; skew 500 ppm
I20260805 22:47:09.655561 7809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:09 dist-test-slave-7lbf krb5kdc[7653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:09.659008 7809 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:47:09.660873 7809 webserver.cc:460] Webserver started at http://127.4.28.1:34219/ using document root <none> and password file <none>
I20260805 22:47:09.661716 7809 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:09.661877 7809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:09.662245 7809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:09.665771 7809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "1264ce9ebd2e4e0cb01c3e853578d486"
format_stamp: "Formatted at 2026-08-05 22:47:09 on dist-test-slave-7lbf"
server_key: "69e719c24be895970b7705b6c9f665fb"
server_key_iv: "bbd028e101424696d35e101eaea552e8"
server_key_version: "encryptionkey@0"
I20260805 22:47:09.666729 7809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "1264ce9ebd2e4e0cb01c3e853578d486"
format_stamp: "Formatted at 2026-08-05 22:47:09 on dist-test-slave-7lbf"
server_key: "69e719c24be895970b7705b6c9f665fb"
server_key_iv: "bbd028e101424696d35e101eaea552e8"
server_key_version: "encryptionkey@0"
I20260805 22:47:09.673036 7809 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260805 22:47:09.677197 7831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:09.678859 7809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:09.679041 7809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "1264ce9ebd2e4e0cb01c3e853578d486"
format_stamp: "Formatted at 2026-08-05 22:47:09 on dist-test-slave-7lbf"
server_key: "69e719c24be895970b7705b6c9f665fb"
server_key_iv: "bbd028e101424696d35e101eaea552e8"
server_key_version: "encryptionkey@0"
I20260805 22:47:09.679253 7809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:09.694240 7809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:09.698844 7809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:09.699141 7809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:09.700191 7809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:09.701730 7809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:09.701813 7809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:09.701939 7809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:09.701998 7809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:09.734397 7809 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:33493
I20260805 22:47:09.734472 7986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:33493 every 8 connection(s)
I20260805 22:47:09.735950 7809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:09.737183 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 7809
I20260805 22:47:09.737409 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:09.738256 4208 external_mini_cluster.cc:1428] Setting key 43cd33e861c2bfbd215d2f9ce3dc4fd1
Aug 05 22:47:09 dist-test-slave-7lbf krb5kdc[7653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:09.762028 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:09.739109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:60877 (local address 127.4.28.62:42895)
0805 22:47:09.739605 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:09.739609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:09.741060 (+ 1451us) server_negotiation.cc:407] Connection header received
0805 22:47:09.743756 (+ 2696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:09.743761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:09.743838 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:09.743938 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:09.746346 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:09.747154 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:09.749653 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:09.750316 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:09.750476 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:09.754644 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:09.754651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:09.754654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:09.754698 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:09.757507 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:09.758833 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:09.758841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:09.758903 (+ 62us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:09.759090 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:09.759754 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:09.759759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:09.759761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:09.760130 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:09.760272 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:09.761385 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0805 22:47:09.761672 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
I20260805 22:47:09.763760 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:09.739876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:42895 (local address 127.4.28.1:60877)
0805 22:47:09.740853 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:09.740918 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:09.743381 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:09.744245 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:09.744258 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:09.745086 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:09.746104 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:09.746139 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:09.747375 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:09.747380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:09.749344 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:09.749364 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:09.750620 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:09.750626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:09.750847 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:09.751581 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:09.751606 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:09.754294 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:09.757782 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:09.757795 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:09.757818 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:09.758591 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:09.759238 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:09.759244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:09.759247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:09.759582 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:09.760295 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:09.760306 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:09.760869 (+ 563us) client_negotiation.cc:769] Sending connection context
0805 22:47:09.761228 (+ 359us) client_negotiation.cc:240] Negotiation successful
0805 22:47:09.761948 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":355,"threads_started":1}
I20260805 22:47:09.765632 7987 heartbeater.cc:347] Connected to a master server at 127.4.28.62:42895
I20260805 22:47:09.766000 7987 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
I20260805 22:47:09.766846 7987 heartbeater.cc:511] Master 127.4.28.62:42895 requested a full tablet report, sending...
I20260805 22:47:09.769400 7717 ts_manager.cc:194] Registered new tserver with Master: 1264ce9ebd2e4e0cb01c3e853578d486 (127.4.28.1:33493)
I20260805 22:47:09.770970 7717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:60877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:47:09.794876 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:42895
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:10.143121 7994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:10.143442 7994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:10.143543 7994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:10.151106 7994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:10.151242 7994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:10.151356 7994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:10.161476 7994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:42895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:10.162883 7994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:10.164255 7994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:10.178167 8004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:10.178298 8001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:10.178269 8000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:10.179188 7994 server_base.cc:1034] running on GCE node
I20260805 22:47:10.179919 7994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:10.180994 7994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:10.182252 7994 hybrid_clock.cc:648] HybridClock initialized: now 1785970030182186 us; error 49 us; skew 500 ppm
I20260805 22:47:10.182691 7994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:10 dist-test-slave-7lbf krb5kdc[7653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970030, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:10.186436 7994 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:47:10.188011 7994 webserver.cc:460] Webserver started at http://127.4.28.2:35029/ using document root <none> and password file <none>
I20260805 22:47:10.188778 7994 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:10.188905 7994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:10.189257 7994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:10.192572 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "ed8e263e10464f3da3b331337f4fea7b"
format_stamp: "Formatted at 2026-08-05 22:47:10 on dist-test-slave-7lbf"
server_key: "5bbd0d0ea2cd01d3c9ff2da6c4f0ad5c"
server_key_iv: "26dea69615eccc230995ce33224c2508"
server_key_version: "encryptionkey@0"
I20260805 22:47:10.193495 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "ed8e263e10464f3da3b331337f4fea7b"
format_stamp: "Formatted at 2026-08-05 22:47:10 on dist-test-slave-7lbf"
server_key: "5bbd0d0ea2cd01d3c9ff2da6c4f0ad5c"
server_key_iv: "26dea69615eccc230995ce33224c2508"
server_key_version: "encryptionkey@0"
I20260805 22:47:10.199653 7994 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260805 22:47:10.203974 8014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:10.205668 7994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:10.205853 7994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "ed8e263e10464f3da3b331337f4fea7b"
format_stamp: "Formatted at 2026-08-05 22:47:10 on dist-test-slave-7lbf"
server_key: "5bbd0d0ea2cd01d3c9ff2da6c4f0ad5c"
server_key_iv: "26dea69615eccc230995ce33224c2508"
server_key_version: "encryptionkey@0"
I20260805 22:47:10.206097 7994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:10.218456 7994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:10.223090 7994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:10.223397 7994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:10.224479 7994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:10.225989 7994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:10.226073 7994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:10.226210 7994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:10.226263 7994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:10.259158 7994 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:41027
I20260805 22:47:10.259196 8174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:41027 every 8 connection(s)
I20260805 22:47:10.260638 7994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:10.269829 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 7994
I20260805 22:47:10.270025 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:47:10.270514 4208 external_mini_cluster.cc:1428] Setting key 7197272488e72bf9e3d5078ceeda8776
Aug 05 22:47:10 dist-test-slave-7lbf krb5kdc[7653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970030, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:10.282979 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:10.263425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:33961 (local address 127.4.28.62:42895)
0805 22:47:10.263877 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:10.263883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:10.265090 (+ 1207us) server_negotiation.cc:407] Connection header received
0805 22:47:10.267093 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:10.267097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:10.267172 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:10.267278 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:10.268898 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:10.269878 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:10.271761 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:10.272274 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:10.272406 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:10.276293 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:10.276302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:10.276306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:10.276354 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:10.279535 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:10.280391 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:10.280397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:10.280400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:10.280566 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:10.280982 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:10.280986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:10.280989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:10.281327 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:10.281448 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:10.282432 (+ 984us) server_negotiation.cc:299] Negotiation successful
0805 22:47:10.282688 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
I20260805 22:47:10.284466 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:10.264040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:42895 (local address 127.4.28.2:33961)
0805 22:47:10.264909 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:10.264958 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:10.266787 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:10.267490 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:10.267511 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:10.268062 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:10.268691 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:10.268712 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:10.270073 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:10.270079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:10.271577 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:10.271601 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:10.272527 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:10.272536 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:10.272830 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:10.273493 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:10.273555 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:10.275975 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:10.279727 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:10.279737 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:10.279754 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:10.280257 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:10.280666 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:10.280669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:10.280671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:10.280872 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:10.281458 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:10.281468 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:10.281976 (+ 508us) client_negotiation.cc:769] Sending connection context
0805 22:47:10.282342 (+ 366us) client_negotiation.cc:240] Negotiation successful
0805 22:47:10.282941 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":363,"threads_started":1}
I20260805 22:47:10.286289 8175 heartbeater.cc:347] Connected to a master server at 127.4.28.62:42895
I20260805 22:47:10.286659 8175 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:10.287451 8175 heartbeater.cc:511] Master 127.4.28.62:42895 requested a full tablet report, sending...
I20260805 22:47:10.289295 7717 ts_manager.cc:194] Registered new tserver with Master: ed8e263e10464f3da3b331337f4fea7b (127.4.28.2:41027)
I20260805 22:47:10.290520 7717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:33961
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:47:10.325156 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:42895
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:10.670485 8182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:10.670825 8182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:10.670970 8182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:10.678491 8182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:10.678619 8182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:10.678769 8182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:10.688977 8182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:42895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:10.690390 8182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:10.691820 8182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:10.705893 8188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:10.705977 8191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:10.705932 8189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:10.711426 8182 server_base.cc:1034] running on GCE node
I20260805 22:47:10.712157 8182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:10.713229 8182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:10.714455 8182 hybrid_clock.cc:648] HybridClock initialized: now 1785970030714372 us; error 78 us; skew 500 ppm
I20260805 22:47:10.714808 8182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:10 dist-test-slave-7lbf krb5kdc[7653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970030, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:10.718156 8182 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:47:10.719906 8182 webserver.cc:460] Webserver started at http://127.4.28.3:42705/ using document root <none> and password file <none>
I20260805 22:47:10.720703 8182 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:10.720827 8182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:10.721163 8182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:10.724331 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "b6ba3ae1682149b982a06d3e30536804"
format_stamp: "Formatted at 2026-08-05 22:47:10 on dist-test-slave-7lbf"
server_key: "24ed18e0dcf9209c5efabfad9d1b8a82"
server_key_iv: "c26615601c94f78f8ce903300cbe9916"
server_key_version: "encryptionkey@0"
I20260805 22:47:10.725173 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "b6ba3ae1682149b982a06d3e30536804"
format_stamp: "Formatted at 2026-08-05 22:47:10 on dist-test-slave-7lbf"
server_key: "24ed18e0dcf9209c5efabfad9d1b8a82"
server_key_iv: "c26615601c94f78f8ce903300cbe9916"
server_key_version: "encryptionkey@0"
I20260805 22:47:10.730834 8182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260805 22:47:10.735016 8204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:10.736862 8182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260805 22:47:10.737087 8182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "b6ba3ae1682149b982a06d3e30536804"
format_stamp: "Formatted at 2026-08-05 22:47:10 on dist-test-slave-7lbf"
server_key: "24ed18e0dcf9209c5efabfad9d1b8a82"
server_key_iv: "c26615601c94f78f8ce903300cbe9916"
server_key_version: "encryptionkey@0"
I20260805 22:47:10.737344 8182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:10.759840 8182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:10.764461 8182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:10.764732 8182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:10.765785 8182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:10.767207 8182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:10.767297 8182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:10.767398 8182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:10.767442 8182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:10.774844 7987 heartbeater.cc:503] Master 127.4.28.62:42895 was elected leader, sending a full tablet report...
I20260805 22:47:10.806694 8182 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:37799
I20260805 22:47:10.806767 8357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:37799 every 8 connection(s)
I20260805 22:47:10.808113 8182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:10.810105 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 8182
I20260805 22:47:10.810298 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:47:10.810838 4208 external_mini_cluster.cc:1428] Setting key 0ec732caf6d30ab674d09587b731a0a8
Aug 05 22:47:10 dist-test-slave-7lbf krb5kdc[7653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970030, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:10.831306 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:10.811518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:60669 (local address 127.4.28.62:42895)
0805 22:47:10.812060 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:10.812067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:10.813296 (+ 1229us) server_negotiation.cc:407] Connection header received
0805 22:47:10.815631 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:10.815636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:10.815733 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:10.815832 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:10.817703 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:10.818522 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:10.820447 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:10.820910 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:10.821033 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:10.824729 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:10.824735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:10.824739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:10.824777 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:10.827960 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:10.828792 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:10.828797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:10.828800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:10.828936 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:10.829389 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:10.829393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:10.829395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:10.829721 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:10.829867 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:10.830668 (+ 801us) server_negotiation.cc:299] Negotiation successful
0805 22:47:10.830923 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":245,"threads_started":1}
I20260805 22:47:10.832310 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:10.812147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:42895 (local address 127.4.28.3:60669)
0805 22:47:10.813087 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:10.813158 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:10.815312 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:10.816168 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:10.816181 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:10.816801 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:10.817466 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:10.817492 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:10.818740 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:10.818746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:10.820287 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:10.820303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:10.821167 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:10.821175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:10.821427 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:10.822160 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:10.822187 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:10.824474 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:10.828121 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:10.828132 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:10.828151 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:10.828656 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:10.829052 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:10.829057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:10.829061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:10.829267 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:10.829851 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:10.829860 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:10.830240 (+ 380us) client_negotiation.cc:769] Sending connection context
0805 22:47:10.830510 (+ 270us) client_negotiation.cc:240] Negotiation successful
0805 22:47:10.831105 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":364,"threads_started":1}
I20260805 22:47:10.833808 8358 heartbeater.cc:347] Connected to a master server at 127.4.28.62:42895
I20260805 22:47:10.834116 8358 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:10.834846 8358 heartbeater.cc:511] Master 127.4.28.62:42895 requested a full tablet report, sending...
I20260805 22:47:10.836462 7717 ts_manager.cc:194] Registered new tserver with Master: b6ba3ae1682149b982a06d3e30536804 (127.4.28.3:37799)
I20260805 22:47:10.837278 7717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:60669
I20260805 22:47:10.839869 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260805 22:47:10.880622 8361 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:10.877872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.4.28.62:42895)
0805 22:47:10.878102 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:10.878107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:10.878547 (+ 440us) server_negotiation.cc:407] Connection header received
0805 22:47:10.878723 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:10.878727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:10.878822 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:10.878927 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:10.880383 (+ 1456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52088: BlockingRecv error: recv got EOF from 127.0.0.1:52088 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 22:47:10.884481 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 7809
W20260805 22:47:10.887786 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 7809: Not authorized: Client connection negotiation failed: client connection to 127.4.28.1:33493: server requires authentication, but client does not have Kerberos credentials available
I20260805 22:47:10.888041 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 7809
I20260805 22:47:10.908391 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 7994
W20260805 22:47:10.911293 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 7994: Not authorized: Client connection negotiation failed: client connection to 127.4.28.2:41027: server requires authentication, but client does not have Kerberos credentials available
I20260805 22:47:10.911432 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 7994
I20260805 22:47:10.935590 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 8182
W20260805 22:47:10.938725 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 8182: Not authorized: Client connection negotiation failed: client connection to 127.4.28.3:37799: server requires authentication, but client does not have Kerberos credentials available
I20260805 22:47:10.938918 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 8182
I20260805 22:47:10.958024 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 7669
I20260805 22:47:10.959460 7762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:11.064357 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 7669
2026-08-05T22:47:11Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3892 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:11 dist-test-slave-7lbf krb5kdc[8382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:11 dist-test-slave-7lbf krb5kdc[8382](info): set up 2 sockets
Aug 05 22:47:11 dist-test-slave-7lbf krb5kdc[8382](info): commencing operation
krb5kdc: starting...
W20260805 22:47:12.251708 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:12 dist-test-slave-7lbf krb5kdc[8382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:47:12.495335 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:42407
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:42407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:12.843406 8399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:12.843750 8399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:12.843874 8399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:12.851450 8399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:12.851579 8399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:12.851634 8399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:12.851716 8399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:12.851773 8399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:12.862039 8399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:42407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:42407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:12.863493 8399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:12.864903 8399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:12.877579 8407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:12.877473 8408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:12.877506 8412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:12.878470 8399 server_base.cc:1034] running on GCE node
I20260805 22:47:12.879305 8399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:12.880867 8399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:12.882102 8399 hybrid_clock.cc:648] HybridClock initialized: now 1785970032882048 us; error 39 us; skew 500 ppm
I20260805 22:47:12.882520 8399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:12 dist-test-slave-7lbf krb5kdc[8382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:12.890138 8399 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:47:12.892033 8399 webserver.cc:460] Webserver started at http://127.4.28.62:41071/ using document root <none> and password file <none>
I20260805 22:47:12.892844 8399 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:12.892969 8399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:12.893266 8399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:12.896657 8399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "b53dfb2088864fcb9d3c79884b2f5a03"
format_stamp: "Formatted at 2026-08-05 22:47:12 on dist-test-slave-7lbf"
server_key: "4a42976b01258f2a9abd8ce335daa29e"
server_key_iv: "966eab47520dc7ebd8deb4af583197f3"
server_key_version: "encryptionkey@0"
I20260805 22:47:12.897619 8399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "b53dfb2088864fcb9d3c79884b2f5a03"
format_stamp: "Formatted at 2026-08-05 22:47:12 on dist-test-slave-7lbf"
server_key: "4a42976b01258f2a9abd8ce335daa29e"
server_key_iv: "966eab47520dc7ebd8deb4af583197f3"
server_key_version: "encryptionkey@0"
I20260805 22:47:12.904188 8399 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260805 22:47:12.908412 8420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:12.910086 8399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260805 22:47:12.910293 8399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "b53dfb2088864fcb9d3c79884b2f5a03"
format_stamp: "Formatted at 2026-08-05 22:47:12 on dist-test-slave-7lbf"
server_key: "4a42976b01258f2a9abd8ce335daa29e"
server_key_iv: "966eab47520dc7ebd8deb4af583197f3"
server_key_version: "encryptionkey@0"
I20260805 22:47:12.910521 8399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:12.941421 8399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:12.946242 8399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:12.946547 8399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:12.965478 8399 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:42407
I20260805 22:47:12.965500 8493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:42407 every 8 connection(s)
I20260805 22:47:12.966979 8399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:12.970870 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 8399
I20260805 22:47:12.971129 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:12.971211 8494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:12.971786 4208 external_mini_cluster.cc:1428] Setting key 6068bd412b0fa500b097a6c91ff088b4
I20260805 22:47:12.986900 8494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03: Bootstrap starting.
Aug 05 22:47:12 dist-test-slave-7lbf krb5kdc[8382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:12.990438 8494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:12.991636 8494 log.cc:824] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:12.995111 8494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03: No bootstrap required, opened a new log
I20260805 22:47:12.996609 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:12.973746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.4.28.62:42407)
0805 22:47:12.974690 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:12.974703 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:12.974753 (+ 50us) server_negotiation.cc:407] Connection header received
0805 22:47:12.976574 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:12.976606 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:12.977102 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:12.977557 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:12.979596 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:12.980985 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:12.982911 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:12.984027 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:12.984203 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:12.988946 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:12.988957 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:12.988976 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:12.989028 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:12.992125 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:12.992993 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:12.993002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:12.993010 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:12.993192 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:12.993681 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:12.993686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:12.993688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:12.994313 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:12.994585 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:12.995048 (+ 463us) server_negotiation.cc:299] Negotiation successful
0805 22:47:12.995537 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":441,"threads_started":1}
I20260805 22:47:13.001155 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:47:13.004478 8494 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dfb2088864fcb9d3c79884b2f5a03" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42407 } }
I20260805 22:47:13.004897 8494 raft_consensus.cc:374] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:13.004977 8494 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53dfb2088864fcb9d3c79884b2f5a03, State: Initialized, Role: FOLLOWER
I20260805 22:47:13.005622 8494 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dfb2088864fcb9d3c79884b2f5a03" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42407 } }
I20260805 22:47:13.005918 8494 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:13.006067 8494 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:13.006237 8494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:13.008620 8494 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dfb2088864fcb9d3c79884b2f5a03" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42407 } }
I20260805 22:47:13.009234 8494 leader_election.cc:302] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53dfb2088864fcb9d3c79884b2f5a03; no voters:
I20260805 22:47:13.010010 8494 leader_election.cc:288] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:13.010134 8499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:13.012812 8494 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:13.013824 8499 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [term 1 LEADER]: Becoming Leader. State: Replica: b53dfb2088864fcb9d3c79884b2f5a03, State: Running, Role: LEADER
I20260805 22:47:13.014755 8499 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dfb2088864fcb9d3c79884b2f5a03" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42407 } }
I20260805 22:47:13.018484 8502 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53dfb2088864fcb9d3c79884b2f5a03. Latest consensus state: current_term: 1 leader_uuid: "b53dfb2088864fcb9d3c79884b2f5a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dfb2088864fcb9d3c79884b2f5a03" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42407 } } }
I20260805 22:47:13.018967 8502 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:13.019676 8501 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b53dfb2088864fcb9d3c79884b2f5a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dfb2088864fcb9d3c79884b2f5a03" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 42407 } } }
I20260805 22:47:13.019997 8501 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:13.019997 8513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:13.039343 8513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:13.066336 8513 catalog_manager.cc:1347] Generated new cluster ID: e3d62f33ef1f4ef8876de319f22a880f
I20260805 22:47:13.066495 8513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:13.084611 8513 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:13.084673 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:13.074833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.4.28.62:42407)
0805 22:47:13.075021 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:13.075027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:13.075597 (+ 570us) server_negotiation.cc:407] Connection header received
0805 22:47:13.075781 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:13.075788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:13.075889 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:13.076013 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:13.077235 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:13.078416 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:13.080224 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:13.080902 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:13.081042 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:13.082405 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:13.082412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:13.082418 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:13.083691 (+ 1273us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:13.083756 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:13.083794 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:13.083941 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:13.084153 (+ 212us) server_negotiation.cc:299] Negotiation successful
0805 22:47:13.084268 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":60}
I20260805 22:47:13.086290 8513 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:13.099078 8513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b53dfb2088864fcb9d3c79884b2f5a03: Generated new TSK 0
I20260805 22:47:13.100312 8513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:47:13.118142 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 8399
I20260805 22:47:13.119498 8489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:13.213572 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 8399
2026-08-05T22:47:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2147 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:13 dist-test-slave-7lbf krb5kdc[8540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:13 dist-test-slave-7lbf krb5kdc[8540](info): set up 2 sockets
Aug 05 22:47:13 dist-test-slave-7lbf krb5kdc[8540](info): commencing operation
krb5kdc: starting...
W20260805 22:47:14.403532 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:14 dist-test-slave-7lbf krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:47:14.655448 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34541
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:34541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:15.005425 8557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:15.005787 8557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:15.005925 8557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:15.014674 8557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:15.014808 8557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:15.014868 8557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:15.014948 8557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:15.015012 8557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:15.025899 8557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:34541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:15.027390 8557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:15.028848 8557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:15.041582 8563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:15.041582 8564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:15.041540 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:15.042608 8557 server_base.cc:1034] running on GCE node
I20260805 22:47:15.043535 8557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:15.045205 8557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:15.046438 8557 hybrid_clock.cc:648] HybridClock initialized: now 1785970035046367 us; error 61 us; skew 500 ppm
I20260805 22:47:15.046840 8557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:15 dist-test-slave-7lbf krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970035, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:15.050170 8557 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:47:15.051839 8557 webserver.cc:460] Webserver started at http://127.4.28.62:40357/ using document root <none> and password file <none>
I20260805 22:47:15.052589 8557 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:15.052716 8557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:15.053047 8557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:15.056341 8557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "ed7f74c3a6ca42c5b26eafaf7284a313"
format_stamp: "Formatted at 2026-08-05 22:47:15 on dist-test-slave-7lbf"
server_key: "7cd9285c917de0ff99c67dc23099eddd"
server_key_iv: "66b9cd383d4657100ce923c4b86d2e13"
server_key_version: "encryptionkey@0"
I20260805 22:47:15.057199 8557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "ed7f74c3a6ca42c5b26eafaf7284a313"
format_stamp: "Formatted at 2026-08-05 22:47:15 on dist-test-slave-7lbf"
server_key: "7cd9285c917de0ff99c67dc23099eddd"
server_key_iv: "66b9cd383d4657100ce923c4b86d2e13"
server_key_version: "encryptionkey@0"
I20260805 22:47:15.062839 8557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260805 22:47:15.066746 8578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:15.068233 8557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:47:15.068434 8557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "ed7f74c3a6ca42c5b26eafaf7284a313"
format_stamp: "Formatted at 2026-08-05 22:47:15 on dist-test-slave-7lbf"
server_key: "7cd9285c917de0ff99c67dc23099eddd"
server_key_iv: "66b9cd383d4657100ce923c4b86d2e13"
server_key_version: "encryptionkey@0"
I20260805 22:47:15.068641 8557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:15.089745 8557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:15.094573 8557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:15.094887 8557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:15.113755 8557 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:34541
I20260805 22:47:15.113794 8660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:34541 every 8 connection(s)
I20260805 22:47:15.115270 8557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:15.119455 8661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:15.120779 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 8557
I20260805 22:47:15.121060 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:15.121969 4208 external_mini_cluster.cc:1428] Setting key 56f30276bb57cad5b3ec57e81ab3c7f7
I20260805 22:47:15.135242 8661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313: Bootstrap starting.
Aug 05 22:47:15 dist-test-slave-7lbf krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:15.139665 8661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:15.140930 8661 log.cc:824] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:15.144641 8661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313: No bootstrap required, opened a new log
I20260805 22:47:15.148620 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:15.124307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.4.28.62:34541)
0805 22:47:15.125205 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:15.125222 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:15.125270 (+ 48us) server_negotiation.cc:407] Connection header received
0805 22:47:15.126970 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:15.127003 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:15.127486 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:15.127873 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:15.129724 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:15.131162 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:15.133131 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:15.134081 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:15.134258 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:15.139509 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:15.139520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:15.139539 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:15.139593 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:15.143352 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:15.144192 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:15.144201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:15.144211 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:15.144457 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:15.144981 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:15.144986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:15.144989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:15.145809 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:15.146154 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:15.146550 (+ 396us) server_negotiation.cc:299] Negotiation successful
0805 22:47:15.147070 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":396,"threads_started":1}
I20260805 22:47:15.154445 8661 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f74c3a6ca42c5b26eafaf7284a313" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34541 } }
I20260805 22:47:15.154839 8661 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:15.154942 8661 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7f74c3a6ca42c5b26eafaf7284a313, State: Initialized, Role: FOLLOWER
I20260805 22:47:15.155612 8661 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f74c3a6ca42c5b26eafaf7284a313" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34541 } }
I20260805 22:47:15.155892 8661 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:15.156047 8661 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:15.156206 8661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:15.158614 8661 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f74c3a6ca42c5b26eafaf7284a313" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34541 } }
I20260805 22:47:15.159184 8661 leader_election.cc:302] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed7f74c3a6ca42c5b26eafaf7284a313; no voters:
I20260805 22:47:15.159837 8661 leader_election.cc:288] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:15.159971 8666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:15.160910 8666 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [term 1 LEADER]: Becoming Leader. State: Replica: ed7f74c3a6ca42c5b26eafaf7284a313, State: Running, Role: LEADER
I20260805 22:47:15.161487 8666 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f74c3a6ca42c5b26eafaf7284a313" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34541 } }
I20260805 22:47:15.162166 8661 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:15.164906 8668 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed7f74c3a6ca42c5b26eafaf7284a313. Latest consensus state: current_term: 1 leader_uuid: "ed7f74c3a6ca42c5b26eafaf7284a313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f74c3a6ca42c5b26eafaf7284a313" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34541 } } }
I20260805 22:47:15.164996 8667 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed7f74c3a6ca42c5b26eafaf7284a313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f74c3a6ca42c5b26eafaf7284a313" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34541 } } }
I20260805 22:47:15.165316 8668 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:15.165357 8667 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:15.169809 8673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:15.175741 8673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:15.186152 8673 catalog_manager.cc:1347] Generated new cluster ID: aea2dc385412418397cfdaec4e23df1a
I20260805 22:47:15.186282 8673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:15.199393 8673 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:15.201473 8673 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:15.221163 8673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed7f74c3a6ca42c5b26eafaf7284a313: Generated new TSK 0
I20260805 22:47:15.222466 8673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:47:15.287565 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34541
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:15.640138 8691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:15.640522 8691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:15.640658 8691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:15.648159 8691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:15.648293 8691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:15.648447 8691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:15.658715 8691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:15.660200 8691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:15.661783 8691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:15.676720 8699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:15.676919 8697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:15.676757 8701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:15.677621 8691 server_base.cc:1034] running on GCE node
I20260805 22:47:15.678328 8691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:15.679374 8691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:15.680624 8691 hybrid_clock.cc:648] HybridClock initialized: now 1785970035680561 us; error 53 us; skew 500 ppm
I20260805 22:47:15.680982 8691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:15 dist-test-slave-7lbf krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970035, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:15.684433 8691 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:47:15.686236 8691 webserver.cc:460] Webserver started at http://127.4.28.1:33971/ using document root <none> and password file <none>
I20260805 22:47:15.687017 8691 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:15.687156 8691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:15.687494 8691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:15.690896 8691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "a32573b2e2bf40a9958b56f10f8ebf05"
format_stamp: "Formatted at 2026-08-05 22:47:15 on dist-test-slave-7lbf"
server_key: "1ede978dd9d6b5417110d0e6c9aa466e"
server_key_iv: "e696f03881bcd7a9a158f98f1f9025b1"
server_key_version: "encryptionkey@0"
I20260805 22:47:15.691829 8691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "a32573b2e2bf40a9958b56f10f8ebf05"
format_stamp: "Formatted at 2026-08-05 22:47:15 on dist-test-slave-7lbf"
server_key: "1ede978dd9d6b5417110d0e6c9aa466e"
server_key_iv: "e696f03881bcd7a9a158f98f1f9025b1"
server_key_version: "encryptionkey@0"
I20260805 22:47:15.697862 8691 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260805 22:47:15.701967 8713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:15.703593 8691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:47:15.703790 8691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "a32573b2e2bf40a9958b56f10f8ebf05"
format_stamp: "Formatted at 2026-08-05 22:47:15 on dist-test-slave-7lbf"
server_key: "1ede978dd9d6b5417110d0e6c9aa466e"
server_key_iv: "e696f03881bcd7a9a158f98f1f9025b1"
server_key_version: "encryptionkey@0"
I20260805 22:47:15.704032 8691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:15.752744 8691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:15.757342 8691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:15.757697 8691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:15.758822 8691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:15.760516 8691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:15.760617 8691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:15.760751 8691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:15.760811 8691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:15.794620 8691 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:43343
I20260805 22:47:15.794709 8889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:43343 every 8 connection(s)
I20260805 22:47:15.796000 8691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:15.803923 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 8691
I20260805 22:47:15.804174 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:15.804873 4208 external_mini_cluster.cc:1428] Setting key 34f4bda7f3fc9f6b5b3afacce3806c44
Aug 05 22:47:15 dist-test-slave-7lbf krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970035, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:15.817863 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:15.798771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:45095 (local address 127.4.28.62:34541)
0805 22:47:15.799271 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:15.799278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:15.800418 (+ 1140us) server_negotiation.cc:407] Connection header received
0805 22:47:15.802329 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:15.802333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:15.802411 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:15.802513 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:15.804384 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:15.805175 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:15.807139 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:15.807549 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:15.807685 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:15.811663 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:15.811669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:15.811672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:15.811712 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:15.814264 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:15.815253 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:15.815258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:15.815260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:15.815413 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:15.815950 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:15.815954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:15.815957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:15.816249 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:15.816404 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:15.817196 (+ 792us) server_negotiation.cc:299] Negotiation successful
0805 22:47:15.817449 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":223,"threads_started":1}
I20260805 22:47:15.818953 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:15.799381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34541 (local address 127.4.28.1:45095)
0805 22:47:15.800233 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:15.800291 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:15.802051 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:15.802723 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:15.802731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:15.803322 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:15.804185 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:15.804217 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:15.805357 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:15.805364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:15.806974 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:15.806991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:15.807823 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:15.807832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:15.808077 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:15.808776 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:15.808812 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:15.811395 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:15.814427 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:15.814438 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:15.814459 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:15.815072 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:15.815563 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:15.815567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:15.815570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:15.815793 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:15.816392 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:15.816401 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:15.816781 (+ 380us) client_negotiation.cc:769] Sending connection context
0805 22:47:15.817043 (+ 262us) client_negotiation.cc:240] Negotiation successful
0805 22:47:15.817693 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":343,"threads_started":1}
I20260805 22:47:15.820734 8890 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34541
I20260805 22:47:15.821094 8890 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:15.821923 8890 heartbeater.cc:511] Master 127.4.28.62:34541 requested a full tablet report, sending...
I20260805 22:47:15.824159 8606 ts_manager.cc:194] Registered new tserver with Master: a32573b2e2bf40a9958b56f10f8ebf05 (127.4.28.1:43343)
I20260805 22:47:15.825748 8606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:45095
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:47:15.857293 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34541
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:16.207309 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:16.207640 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:16.207739 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:16.215485 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:16.215622 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:16.215773 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:16.226326 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:16.227789 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:16.229251 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:16.243574 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:16.243592 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:16.244272 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:16.244270 8902 server_base.cc:1034] running on GCE node
I20260805 22:47:16.245061 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:16.246201 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:16.247442 8902 hybrid_clock.cc:648] HybridClock initialized: now 1785970036247373 us; error 58 us; skew 500 ppm
I20260805 22:47:16.247812 8902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:16 dist-test-slave-7lbf krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:16.251356 8902 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:47:16.253041 8902 webserver.cc:460] Webserver started at http://127.4.28.2:43249/ using document root <none> and password file <none>
I20260805 22:47:16.253883 8902 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:16.254004 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:16.254341 8902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:16.257443 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "2438a8d0a919432b8f400d483522cacd"
format_stamp: "Formatted at 2026-08-05 22:47:16 on dist-test-slave-7lbf"
server_key: "53e9036ef0ae227d65a79943d3acc900"
server_key_iv: "a3cdcfd6074c16d56f12f6079971c63c"
server_key_version: "encryptionkey@0"
I20260805 22:47:16.258330 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "2438a8d0a919432b8f400d483522cacd"
format_stamp: "Formatted at 2026-08-05 22:47:16 on dist-test-slave-7lbf"
server_key: "53e9036ef0ae227d65a79943d3acc900"
server_key_iv: "a3cdcfd6074c16d56f12f6079971c63c"
server_key_version: "encryptionkey@0"
I20260805 22:47:16.264194 8902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260805 22:47:16.268280 8920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:16.270102 8902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260805 22:47:16.270313 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "2438a8d0a919432b8f400d483522cacd"
format_stamp: "Formatted at 2026-08-05 22:47:16 on dist-test-slave-7lbf"
server_key: "53e9036ef0ae227d65a79943d3acc900"
server_key_iv: "a3cdcfd6074c16d56f12f6079971c63c"
server_key_version: "encryptionkey@0"
I20260805 22:47:16.270535 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:16.295032 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:16.299742 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:16.300062 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:16.301167 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:16.302762 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:16.302852 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:16.302991 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:16.303049 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:16.335093 8902 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:37467
I20260805 22:47:16.335166 9091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:37467 every 8 connection(s)
I20260805 22:47:16.336431 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:16.344358 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 8902
I20260805 22:47:16.344630 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:47:16.345103 4208 external_mini_cluster.cc:1428] Setting key 79c32944da8408574f8db369f986e32a
Aug 05 22:47:16 dist-test-slave-7lbf krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:16.359045 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:16.339140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:49879 (local address 127.4.28.62:34541)
0805 22:47:16.339611 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:16.339617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:16.340694 (+ 1077us) server_negotiation.cc:407] Connection header received
0805 22:47:16.342673 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:16.342678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:16.342755 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:16.342863 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:16.344862 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.345797 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.347920 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.348368 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.348477 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:16.352882 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:16.352888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:16.352892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:16.352933 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:16.355652 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:16.356477 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:16.356481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:16.356484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:16.356627 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:16.357102 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:16.357106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:16.357108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:16.357414 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:16.357588 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:16.358405 (+ 817us) server_negotiation.cc:299] Negotiation successful
0805 22:47:16.358656 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260805 22:47:16.360247 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:16.339765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34541 (local address 127.4.28.2:49879)
0805 22:47:16.340511 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:16.340565 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:16.342363 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:16.343094 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:16.343103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:16.343680 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:16.344543 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:16.344577 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.346017 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.346031 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:16.347690 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:16.347709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.348628 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.348635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:16.348952 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:16.349886 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:16.349916 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:16.352573 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:16.355835 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:16.355852 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:16.355868 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:16.356343 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:16.356751 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:16.356756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:16.356759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:16.356977 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:16.357586 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:16.357596 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:16.357989 (+ 393us) client_negotiation.cc:769] Sending connection context
0805 22:47:16.358276 (+ 287us) client_negotiation.cc:240] Negotiation successful
0805 22:47:16.358892 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":282,"threads_started":1}
I20260805 22:47:16.361914 9092 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34541
I20260805 22:47:16.362258 9092 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:16.363096 9092 heartbeater.cc:511] Master 127.4.28.62:34541 requested a full tablet report, sending...
I20260805 22:47:16.364794 8606 ts_manager.cc:194] Registered new tserver with Master: 2438a8d0a919432b8f400d483522cacd (127.4.28.2:37467)
I20260805 22:47:16.366039 8606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:49879
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:47:16.398936 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34541
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:16.753728 9099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:16.754061 9099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:16.754171 9099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:16.761832 9099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:16.761972 9099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:16.762099 9099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:16.772826 9099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:16.774405 9099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:16.775934 9099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:16.799010 9109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:16.799769 9111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:16.800168 9099 server_base.cc:1034] running on GCE node
W20260805 22:47:16.800838 9108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:16.801676 9099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:16.802692 9099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:16.803960 9099 hybrid_clock.cc:648] HybridClock initialized: now 1785970036803893 us; error 54 us; skew 500 ppm
I20260805 22:47:16.804352 9099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:16 dist-test-slave-7lbf krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:16.807897 9099 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:47:16.809608 9099 webserver.cc:460] Webserver started at http://127.4.28.3:40965/ using document root <none> and password file <none>
I20260805 22:47:16.810350 9099 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:16.810480 9099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:16.810813 9099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:16.814054 9099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "fcbcf2078370407d8ac6814aad3f7186"
format_stamp: "Formatted at 2026-08-05 22:47:16 on dist-test-slave-7lbf"
server_key: "ea66e6a70bb020cd48bab1773cb2fb5b"
server_key_iv: "6470f62adeb09ca9b311678b95fe0523"
server_key_version: "encryptionkey@0"
I20260805 22:47:16.814914 9099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "fcbcf2078370407d8ac6814aad3f7186"
format_stamp: "Formatted at 2026-08-05 22:47:16 on dist-test-slave-7lbf"
server_key: "ea66e6a70bb020cd48bab1773cb2fb5b"
server_key_iv: "6470f62adeb09ca9b311678b95fe0523"
server_key_version: "encryptionkey@0"
I20260805 22:47:16.820547 9099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260805 22:47:16.824563 9125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:16.826421 9099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:47:16.826615 9099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "fcbcf2078370407d8ac6814aad3f7186"
format_stamp: "Formatted at 2026-08-05 22:47:16 on dist-test-slave-7lbf"
server_key: "ea66e6a70bb020cd48bab1773cb2fb5b"
server_key_iv: "6470f62adeb09ca9b311678b95fe0523"
server_key_version: "encryptionkey@0"
I20260805 22:47:16.826843 9099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:16.828905 8890 heartbeater.cc:503] Master 127.4.28.62:34541 was elected leader, sending a full tablet report...
I20260805 22:47:16.841274 9099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:16.846171 9099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:16.846485 9099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:16.847527 9099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:16.849043 9099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:16.849123 9099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:16.849228 9099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:16.849284 9099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:16.882676 9099 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:44253
I20260805 22:47:16.882725 9285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:44253 every 8 connection(s)
I20260805 22:47:16.884085 9099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:16.886358 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 9099
I20260805 22:47:16.886596 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:47:16.887156 4208 external_mini_cluster.cc:1428] Setting key c04ccc8d219a0ae762909b5d1698d171
Aug 05 22:47:16 dist-test-slave-7lbf krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970036, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:16.907377 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:16.887586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:32935 (local address 127.4.28.62:34541)
0805 22:47:16.888147 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:16.888153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:16.889456 (+ 1303us) server_negotiation.cc:407] Connection header received
0805 22:47:16.891512 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:16.891517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:16.891625 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:16.891726 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:16.893347 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.894230 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.895942 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.896363 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.896485 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:16.900424 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:16.900432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:16.900437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:16.900481 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:16.904013 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:16.904845 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:16.904848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:16.904851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:16.905021 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:16.905577 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:16.905583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:16.905587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:16.905889 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:16.906054 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:16.906859 (+ 805us) server_negotiation.cc:299] Negotiation successful
0805 22:47:16.907089 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":249,"threads_started":1}
I20260805 22:47:16.908520 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:16.888314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34541 (local address 127.4.28.3:32935)
0805 22:47:16.889256 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:16.889308 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:16.891188 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:16.891944 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:16.891954 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:16.892535 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:16.893170 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:16.893190 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.894421 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.894425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:16.895784 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:16.895800 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.896626 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.896633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:16.896867 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:16.897612 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:16.897643 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:16.900109 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:16.904189 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:16.904199 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:16.904216 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:16.904738 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:16.905165 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:16.905169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:16.905172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:16.905421 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:16.906056 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:16.906063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:16.906491 (+ 428us) client_negotiation.cc:769] Sending connection context
0805 22:47:16.906779 (+ 288us) client_negotiation.cc:240] Negotiation successful
0805 22:47:16.907256 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":392,"threads_started":1}
I20260805 22:47:16.910144 9286 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34541
I20260805 22:47:16.910471 9286 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:16.911274 9286 heartbeater.cc:511] Master 127.4.28.62:34541 requested a full tablet report, sending...
I20260805 22:47:16.912920 8606 ts_manager.cc:194] Registered new tserver with Master: fcbcf2078370407d8ac6814aad3f7186 (127.4.28.3:44253)
I20260805 22:47:16.913908 8606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:32935
I20260805 22:47:16.916561 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 22:47:16 dist-test-slave-7lbf krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 05 22:47:16 dist-test-slave-7lbf krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:16.976313 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:16.961167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.4.28.62:34541)
0805 22:47:16.961380 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:16.961386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:16.961878 (+ 492us) server_negotiation.cc:407] Connection header received
0805 22:47:16.962004 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:16.962010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:16.962100 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:16.962198 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:16.963678 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.964535 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.965926 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.966379 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.966494 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:16.970530 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:16.970536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:16.970554 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:16.970600 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:16.973540 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:16.974306 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:16.974311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:16.974313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:16.974462 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:16.974854 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:16.974858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:16.974860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:16.975209 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:16.975401 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:16.975698 (+ 297us) server_negotiation.cc:299] Negotiation successful
0805 22:47:16.976004 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260805 22:47:16.976890 8606 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54758
I20260805 22:47:16.978606 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 8691
Aug 05 22:47:16 dist-test-slave-7lbf krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:47:16.996382 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:16.980216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.4.28.1:43343)
0805 22:47:16.980865 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:16.980873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:16.980901 (+ 28us) server_negotiation.cc:407] Connection header received
0805 22:47:16.980962 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:16.980969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:16.981198 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:16.981331 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:16.982687 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.983799 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.985357 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:16.986047 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:16.986168 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:16.990396 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:16.990413 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:16.990433 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:16.990480 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:16.993495 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:16.994356 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:16.994364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:16.994367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:16.994513 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:16.994846 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:16.994849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:16.994851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:16.995242 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:16.995506 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:16.995855 (+ 349us) server_negotiation.cc:299] Negotiation successful
0805 22:47:16.996088 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":300,"threads_started":1}
W20260805 22:47:16.997814 8883 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36002
W20260805 22:47:16.999070 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 8691: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:47:16.999271 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 8691
I20260805 22:47:17.023207 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 8902
Aug 05 22:47:17 dist-test-slave-7lbf krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:47:17.041249 9306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:17.024221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.4.28.2:37467)
0805 22:47:17.024849 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:17.024857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:17.024886 (+ 29us) server_negotiation.cc:407] Connection header received
0805 22:47:17.024965 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:17.024972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:17.025198 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:17.025337 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:17.026820 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:17.027910 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:17.029358 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:17.030081 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:17.030200 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:17.034664 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:17.034680 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:17.034701 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:17.034763 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:17.037894 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:17.038729 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:17.038739 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:17.038743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:17.038944 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:17.039330 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:17.039333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:17.039336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:17.039745 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:17.040005 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:17.040497 (+ 492us) server_negotiation.cc:299] Negotiation successful
0805 22:47:17.040873 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":305,"threads_started":1}
W20260805 22:47:17.042829 9087 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50196
W20260805 22:47:17.044176 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 8902: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:47:17.044389 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 8902
I20260805 22:47:17.069819 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 9099
Aug 05 22:47:17 dist-test-slave-7lbf krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970036, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:47:17.088068 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:17.071016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.4.28.3:44253)
0805 22:47:17.071688 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:17.071696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:17.071729 (+ 33us) server_negotiation.cc:407] Connection header received
0805 22:47:17.071798 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:17.071805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:17.072106 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:17.072276 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:17.073962 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:17.075053 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:17.076432 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:17.077062 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:17.077180 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:17.081824 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:17.081841 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:17.081869 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:17.081917 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:17.084972 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:17.085961 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:17.085970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:17.085973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:17.086137 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:17.086470 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:17.086473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:17.086476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:17.086928 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:17.087198 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:17.087454 (+ 256us) server_negotiation.cc:299] Negotiation successful
0805 22:47:17.087688 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":324,"threads_started":1}
W20260805 22:47:17.089582 9281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59136
W20260805 22:47:17.090853 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 9099: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:47:17.091010 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 9099
I20260805 22:47:17.114828 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 8557
I20260805 22:47:17.116307 8656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:17.222430 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 8557
2026-08-05T22:47:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4010 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:17 dist-test-slave-7lbf krb5kdc[9313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:17 dist-test-slave-7lbf krb5kdc[9313](info): set up 2 sockets
Aug 05 22:47:17 dist-test-slave-7lbf krb5kdc[9313](info): commencing operation
krb5kdc: starting...
W20260805 22:47:18.415055 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:18 dist-test-slave-7lbf krb5kdc[9313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:47:18.661566 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37257
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:37257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:19.039919 9333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:19.040302 9333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:19.040405 9333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:19.047848 9333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:19.047992 9333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:19.048041 9333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:19.048098 9333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:19.048147 9333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:19.058374 9333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:37257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:19.059769 9333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:19.061156 9333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:19.073262 9341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:19.073540 9346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:19.073678 9343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:19.074225 9333 server_base.cc:1034] running on GCE node
I20260805 22:47:19.075060 9333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:19.076617 9333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:19.077881 9333 hybrid_clock.cc:648] HybridClock initialized: now 1785970039077781 us; error 39 us; skew 500 ppm
I20260805 22:47:19.078255 9333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:19 dist-test-slave-7lbf krb5kdc[9313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970039, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:19.086031 9333 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:47:19.087767 9333 webserver.cc:460] Webserver started at http://127.4.28.62:43351/ using document root <none> and password file <none>
I20260805 22:47:19.088562 9333 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:19.088701 9333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:19.089052 9333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:19.092574 9333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "ccba1937fce8445fa8b3fd47b4c71020"
format_stamp: "Formatted at 2026-08-05 22:47:19 on dist-test-slave-7lbf"
server_key: "333c718748284111cf4000b018907b85"
server_key_iv: "a5775e8f7739b55f00dc0370564cec63"
server_key_version: "encryptionkey@0"
I20260805 22:47:19.093483 9333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "ccba1937fce8445fa8b3fd47b4c71020"
format_stamp: "Formatted at 2026-08-05 22:47:19 on dist-test-slave-7lbf"
server_key: "333c718748284111cf4000b018907b85"
server_key_iv: "a5775e8f7739b55f00dc0370564cec63"
server_key_version: "encryptionkey@0"
I20260805 22:47:19.099375 9333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260805 22:47:19.103402 9357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:19.105059 9333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 22:47:19.105254 9333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "ccba1937fce8445fa8b3fd47b4c71020"
format_stamp: "Formatted at 2026-08-05 22:47:19 on dist-test-slave-7lbf"
server_key: "333c718748284111cf4000b018907b85"
server_key_iv: "a5775e8f7739b55f00dc0370564cec63"
server_key_version: "encryptionkey@0"
I20260805 22:47:19.105484 9333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:19.123298 9333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:19.127866 9333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:19.128157 9333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:19.146183 9333 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:37257
I20260805 22:47:19.146212 9426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:37257 every 8 connection(s)
I20260805 22:47:19.147789 9333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:19.151892 9427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:19.157593 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 9333
I20260805 22:47:19.157817 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:19.158574 4208 external_mini_cluster.cc:1428] Setting key 19165bad62026b3be56a2a9a32ba51af
I20260805 22:47:19.163636 9427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020: Bootstrap starting.
I20260805 22:47:19.167642 9427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:19.169004 9427 log.cc:824] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:19.172689 9427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020: No bootstrap required, opened a new log
Aug 05 22:47:19 dist-test-slave-7lbf krb5kdc[9313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:19.180931 9427 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba1937fce8445fa8b3fd47b4c71020" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37257 } }
I20260805 22:47:19.181365 9427 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:19.181502 9427 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccba1937fce8445fa8b3fd47b4c71020, State: Initialized, Role: FOLLOWER
I20260805 22:47:19.182260 9427 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba1937fce8445fa8b3fd47b4c71020" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37257 } }
I20260805 22:47:19.182571 9427 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:19.182717 9427 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:19.182948 9427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:19.183674 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:19.160895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33442 (local address 127.4.28.62:37257)
0805 22:47:19.161813 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:19.161826 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:19.161874 (+ 48us) server_negotiation.cc:407] Connection header received
0805 22:47:19.163787 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:19.163833 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:19.164299 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:19.164662 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:19.166636 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:19.168117 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:19.169982 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:19.170955 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:19.171099 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:19.175731 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:19.175741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:19.175765 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:19.175814 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:19.179026 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:19.179838 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:19.179846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:19.179853 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:19.180061 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:19.180480 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:19.180484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:19.180487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:19.181108 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:19.181403 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:19.181927 (+ 524us) server_negotiation.cc:299] Negotiation successful
0805 22:47:19.182402 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":403,"threads_started":1}
I20260805 22:47:19.186115 9427 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba1937fce8445fa8b3fd47b4c71020" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37257 } }
I20260805 22:47:19.186702 9427 leader_election.cc:302] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccba1937fce8445fa8b3fd47b4c71020; no voters:
I20260805 22:47:19.187417 9427 leader_election.cc:288] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:19.187561 9432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:19.188215 9432 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [term 1 LEADER]: Becoming Leader. State: Replica: ccba1937fce8445fa8b3fd47b4c71020, State: Running, Role: LEADER
I20260805 22:47:19.188817 9432 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba1937fce8445fa8b3fd47b4c71020" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37257 } }
I20260805 22:47:19.189674 9427 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:19.191778 9433 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccba1937fce8445fa8b3fd47b4c71020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba1937fce8445fa8b3fd47b4c71020" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37257 } } }
I20260805 22:47:19.192240 9434 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccba1937fce8445fa8b3fd47b4c71020. Latest consensus state: current_term: 1 leader_uuid: "ccba1937fce8445fa8b3fd47b4c71020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccba1937fce8445fa8b3fd47b4c71020" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37257 } } }
I20260805 22:47:19.192718 9434 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:19.192718 9433 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:19.194564 9442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:19.200488 9442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:19.211551 9442 catalog_manager.cc:1347] Generated new cluster ID: 0069ab4ec24e46dda2f9b23540dabe7a
I20260805 22:47:19.211697 9442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:19.226311 9442 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:19.227874 9442 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:19.236058 9442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccba1937fce8445fa8b3fd47b4c71020: Generated new TSK 0
I20260805 22:47:19.237056 9442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:47:19.299367 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37257
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:19.653831 9458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:19.654166 9458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:19.654295 9458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:19.661790 9458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:19.661926 9458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:19.662093 9458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:19.672705 9458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:19.674269 9458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:19.675678 9458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:19.689440 9468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:19.689488 9465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:19.689576 9464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:19.690826 9458 server_base.cc:1034] running on GCE node
I20260805 22:47:19.691550 9458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:19.692688 9458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:19.693920 9458 hybrid_clock.cc:648] HybridClock initialized: now 1785970039693843 us; error 75 us; skew 500 ppm
I20260805 22:47:19.694358 9458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:19 dist-test-slave-7lbf krb5kdc[9313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970039, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:19.697758 9458 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:47:19.699373 9458 webserver.cc:460] Webserver started at http://127.4.28.1:45693/ using document root <none> and password file <none>
I20260805 22:47:19.700122 9458 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:19.700246 9458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:19.700584 9458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:19.703986 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "8b30e67e1c7e4eafa66766324c56e915"
format_stamp: "Formatted at 2026-08-05 22:47:19 on dist-test-slave-7lbf"
server_key: "215090337dbb4cb36b885056db1cfb04"
server_key_iv: "79abab7ddbf7726aa341e09815354137"
server_key_version: "encryptionkey@0"
I20260805 22:47:19.704876 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "8b30e67e1c7e4eafa66766324c56e915"
format_stamp: "Formatted at 2026-08-05 22:47:19 on dist-test-slave-7lbf"
server_key: "215090337dbb4cb36b885056db1cfb04"
server_key_iv: "79abab7ddbf7726aa341e09815354137"
server_key_version: "encryptionkey@0"
I20260805 22:47:19.711091 9458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260805 22:47:19.715054 9481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:19.716637 9458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:19.716830 9458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "8b30e67e1c7e4eafa66766324c56e915"
format_stamp: "Formatted at 2026-08-05 22:47:19 on dist-test-slave-7lbf"
server_key: "215090337dbb4cb36b885056db1cfb04"
server_key_iv: "79abab7ddbf7726aa341e09815354137"
server_key_version: "encryptionkey@0"
I20260805 22:47:19.717100 9458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:19.746186 9458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:19.750891 9458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:19.751200 9458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:19.752333 9458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:19.753885 9458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:19.753968 9458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:19.754101 9458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:19.754154 9458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:19.787578 9458 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:35239
I20260805 22:47:19.787624 9634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:35239 every 8 connection(s)
I20260805 22:47:19.788904 9458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:19.795930 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 9458
I20260805 22:47:19.796132 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:19.796795 4208 external_mini_cluster.cc:1428] Setting key 0b7aba195791669941a27a7cf136d12e
Aug 05 22:47:19 dist-test-slave-7lbf krb5kdc[9313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970039, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:19.811662 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:19.791591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:56337 (local address 127.4.28.62:37257)
0805 22:47:19.792059 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:19.792064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:19.793074 (+ 1010us) server_negotiation.cc:407] Connection header received
0805 22:47:19.795184 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:19.795190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:19.795286 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:19.795390 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:19.797231 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:19.798204 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:19.800586 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:19.801010 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:19.801131 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:19.805453 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:19.805459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:19.805462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:19.805501 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:19.808056 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:19.808913 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:19.808918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:19.808920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:19.809080 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:19.809693 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:19.809699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:19.809702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:19.810113 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:19.810294 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:19.811058 (+ 764us) server_negotiation.cc:299] Negotiation successful
0805 22:47:19.811302 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
I20260805 22:47:19.812809 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:19.792120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37257 (local address 127.4.28.1:56337)
0805 22:47:19.792889 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:19.792932 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:19.794858 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:19.795691 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:19.795701 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:19.796343 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:19.797025 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:19.797057 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:19.798423 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:19.798428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:19.800396 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:19.800416 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:19.801275 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:19.801283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:19.801646 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:19.802429 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:19.802461 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:19.805168 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:19.808246 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:19.808258 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:19.808277 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:19.808767 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:19.809201 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:19.809207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:19.809211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:19.809507 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:19.810252 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:19.810259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:19.810640 (+ 381us) client_negotiation.cc:769] Sending connection context
0805 22:47:19.810907 (+ 267us) client_negotiation.cc:240] Negotiation successful
0805 22:47:19.811415 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":292,"threads_started":1}
I20260805 22:47:19.814699 9635 heartbeater.cc:347] Connected to a master server at 127.4.28.62:37257
I20260805 22:47:19.815055 9635 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:19.816231 9635 heartbeater.cc:511] Master 127.4.28.62:37257 requested a full tablet report, sending...
I20260805 22:47:19.819051 9382 ts_manager.cc:194] Registered new tserver with Master: 8b30e67e1c7e4eafa66766324c56e915 (127.4.28.1:35239)
I20260805 22:47:19.821271 9382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:56337
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:47:19.852797 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37257
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:20.201911 9643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:20.202227 9643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:20.202337 9643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:20.210084 9643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:20.210340 9643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:20.210493 9643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:20.221624 9643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:20.223065 9643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:20.224524 9643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:20.239089 9650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:20.239017 9652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:20.239054 9649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:20.239984 9643 server_base.cc:1034] running on GCE node
I20260805 22:47:20.240658 9643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:20.241703 9643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:20.242946 9643 hybrid_clock.cc:648] HybridClock initialized: now 1785970040242882 us; error 42 us; skew 500 ppm
I20260805 22:47:20.243304 9643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:20 dist-test-slave-7lbf krb5kdc[9313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970040, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:20.246954 9643 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:47:20.248813 9643 webserver.cc:460] Webserver started at http://127.4.28.2:45337/ using document root <none> and password file <none>
I20260805 22:47:20.249722 9643 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:20.249876 9643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:20.250252 9643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:20.253773 9643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "784f8d96f4274915bd88bbd00da1286c"
format_stamp: "Formatted at 2026-08-05 22:47:20 on dist-test-slave-7lbf"
server_key: "eeb312713851bfaf736c78bcf6169303"
server_key_iv: "698b4a0cff95fe488bf95e49d1f8880c"
server_key_version: "encryptionkey@0"
I20260805 22:47:20.254796 9643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "784f8d96f4274915bd88bbd00da1286c"
format_stamp: "Formatted at 2026-08-05 22:47:20 on dist-test-slave-7lbf"
server_key: "eeb312713851bfaf736c78bcf6169303"
server_key_iv: "698b4a0cff95fe488bf95e49d1f8880c"
server_key_version: "encryptionkey@0"
I20260805 22:47:20.261376 9643 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260805 22:47:20.265648 9659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:20.267493 9643 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:47:20.267717 9643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "784f8d96f4274915bd88bbd00da1286c"
format_stamp: "Formatted at 2026-08-05 22:47:20 on dist-test-slave-7lbf"
server_key: "eeb312713851bfaf736c78bcf6169303"
server_key_iv: "698b4a0cff95fe488bf95e49d1f8880c"
server_key_version: "encryptionkey@0"
I20260805 22:47:20.267963 9643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:20.290359 9643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:20.295053 9643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:20.295362 9643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:20.296389 9643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:20.297856 9643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:20.297940 9643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:20.298074 9643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:20.298127 9643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:20.330093 9643 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:35825
I20260805 22:47:20.330165 9809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:35825 every 8 connection(s)
I20260805 22:47:20.331482 9643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:20.338428 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 9643
I20260805 22:47:20.338598 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:47:20.339066 4208 external_mini_cluster.cc:1428] Setting key c499385b127b958559465296dc3cb929
Aug 05 22:47:20 dist-test-slave-7lbf krb5kdc[9313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970040, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:20.354279 9812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:20.334304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:56317 (local address 127.4.28.62:37257)
0805 22:47:20.334775 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:20.334780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:20.335758 (+ 978us) server_negotiation.cc:407] Connection header received
0805 22:47:20.337756 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:20.337762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:20.337859 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:20.337962 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:20.339920 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.340849 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.342903 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.343381 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.343509 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:20.347450 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:20.347456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:20.347460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:20.347501 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:20.350530 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:20.351527 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:20.351532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:20.351535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:20.351706 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:20.352239 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:20.352244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:20.352247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:20.352573 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:20.352722 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:20.353626 (+ 904us) server_negotiation.cc:299] Negotiation successful
0805 22:47:20.353899 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":253,"threads_started":1}
I20260805 22:47:20.355410 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:20.334815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37257 (local address 127.4.28.2:56317)
0805 22:47:20.335571 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:20.335626 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:20.337393 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:20.338272 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:20.338281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:20.338925 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:20.339642 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:20.339673 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.341180 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.341186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:20.342713 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:20.342730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.343666 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.343675 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:20.343924 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:20.344658 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:20.344689 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:20.347132 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:20.350769 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:20.350780 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:20.350798 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:20.351345 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:20.351848 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:20.351859 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:20.351861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:20.352095 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:20.352730 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:20.352740 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:20.353164 (+ 424us) client_negotiation.cc:769] Sending connection context
0805 22:47:20.353442 (+ 278us) client_negotiation.cc:240] Negotiation successful
0805 22:47:20.354121 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":286,"threads_started":1}
I20260805 22:47:20.357215 9810 heartbeater.cc:347] Connected to a master server at 127.4.28.62:37257
I20260805 22:47:20.357576 9810 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:20.358457 9810 heartbeater.cc:511] Master 127.4.28.62:37257 requested a full tablet report, sending...
I20260805 22:47:20.360229 9382 ts_manager.cc:194] Registered new tserver with Master: 784f8d96f4274915bd88bbd00da1286c (127.4.28.2:35825)
I20260805 22:47:20.361435 9382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:56317
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:47:20.394941 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37257
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:20.745740 9819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:20.746068 9819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:20.746208 9819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:20.753780 9819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:20.753921 9819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:20.754079 9819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:20.764274 9819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:20.765736 9819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:20.767184 9819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:20.781757 9826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:20.781838 9830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:20.781788 9827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:20.782712 9819 server_base.cc:1034] running on GCE node
I20260805 22:47:20.783437 9819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:20.784524 9819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:20.785771 9819 hybrid_clock.cc:648] HybridClock initialized: now 1785970040785704 us; error 52 us; skew 500 ppm
I20260805 22:47:20.786291 9819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:20 dist-test-slave-7lbf krb5kdc[9313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970040, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:20.790982 9819 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:47:20.795183 9819 webserver.cc:460] Webserver started at http://127.4.28.3:37167/ using document root <none> and password file <none>
I20260805 22:47:20.796134 9819 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:20.796264 9819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:20.796597 9819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:20.799952 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "43df0d092f744ac4842080618d8194ce"
format_stamp: "Formatted at 2026-08-05 22:47:20 on dist-test-slave-7lbf"
server_key: "ca79639f85125436f315351652313555"
server_key_iv: "3f74b40cc86a4d3f9b057365cd62d05f"
server_key_version: "encryptionkey@0"
I20260805 22:47:20.800846 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "43df0d092f744ac4842080618d8194ce"
format_stamp: "Formatted at 2026-08-05 22:47:20 on dist-test-slave-7lbf"
server_key: "ca79639f85125436f315351652313555"
server_key_iv: "3f74b40cc86a4d3f9b057365cd62d05f"
server_key_version: "encryptionkey@0"
I20260805 22:47:20.806766 9819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260805 22:47:20.810804 9842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:20.812455 9819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260805 22:47:20.812672 9819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "43df0d092f744ac4842080618d8194ce"
format_stamp: "Formatted at 2026-08-05 22:47:20 on dist-test-slave-7lbf"
server_key: "ca79639f85125436f315351652313555"
server_key_iv: "3f74b40cc86a4d3f9b057365cd62d05f"
server_key_version: "encryptionkey@0"
I20260805 22:47:20.812894 9819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:20.824831 9635 heartbeater.cc:503] Master 127.4.28.62:37257 was elected leader, sending a full tablet report...
I20260805 22:47:20.837371 9819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:20.842116 9819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:20.842430 9819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:20.843602 9819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:20.845075 9819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:20.845162 9819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:20.845293 9819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:20.845355 9819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:20.878166 9819 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:40969
I20260805 22:47:20.878247 10011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:40969 every 8 connection(s)
I20260805 22:47:20.879535 9819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:20.881161 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 9819
I20260805 22:47:20.881326 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:47:20.881922 4208 external_mini_cluster.cc:1428] Setting key e05349b5af387e1cd93f1f3c781b1f7f
Aug 05 22:47:20 dist-test-slave-7lbf krb5kdc[9313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970040, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:20.902446 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:20.882339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:60341 (local address 127.4.28.62:37257)
0805 22:47:20.882840 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:20.882852 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:20.884571 (+ 1719us) server_negotiation.cc:407] Connection header received
0805 22:47:20.886636 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:20.886641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:20.886732 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:20.886832 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:20.888507 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.889298 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.891043 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.891466 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.891589 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:20.895678 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:20.895687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:20.895693 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:20.895750 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:20.899092 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:20.899930 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:20.899935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:20.899938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:20.900070 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:20.900538 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:20.900542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:20.900544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:20.900832 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:20.900974 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:20.901802 (+ 828us) server_negotiation.cc:299] Negotiation successful
0805 22:47:20.902065 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":231,"threads_started":1}
I20260805 22:47:20.903648 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:20.882952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37257 (local address 127.4.28.3:60341)
0805 22:47:20.884400 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:20.884453 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:20.886331 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:20.887115 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:20.887126 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:20.887696 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:20.888314 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:20.888333 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.889460 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.889464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:20.890890 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:20.890905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.891716 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.891724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:20.892021 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:20.892799 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:20.892836 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:20.895374 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:20.899258 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:20.899267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:20.899284 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:20.899778 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:20.900184 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:20.900188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:20.900190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:20.900415 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:20.900984 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:20.900994 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:20.901360 (+ 366us) client_negotiation.cc:769] Sending connection context
0805 22:47:20.901675 (+ 315us) client_negotiation.cc:240] Negotiation successful
0805 22:47:20.902257 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":865,"threads_started":1}
I20260805 22:47:20.905180 10012 heartbeater.cc:347] Connected to a master server at 127.4.28.62:37257
I20260805 22:47:20.905478 10012 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:20.906263 10012 heartbeater.cc:511] Master 127.4.28.62:37257 requested a full tablet report, sending...
I20260805 22:47:20.907814 9382 ts_manager.cc:194] Registered new tserver with Master: 43df0d092f744ac4842080618d8194ce (127.4.28.3:40969)
I20260805 22:47:20.908819 9382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:60341
I20260805 22:47:20.910902 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 22:47:20 dist-test-slave-7lbf krb5kdc[9313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 22:47:20 dist-test-slave-7lbf krb5kdc[9313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:47:20.971926 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:20.955272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.4.28.1:35239)
0805 22:47:20.956119 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:20.956127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:20.956156 (+ 29us) server_negotiation.cc:407] Connection header received
0805 22:47:20.956225 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:20.956232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:20.956457 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:20.956610 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:20.958101 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.959193 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.960513 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.961121 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.961226 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:20.965509 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:20.965578 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:20.965600 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:20.965648 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:20.968825 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:20.969665 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:20.969674 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:20.969677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:20.969823 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:20.970233 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:20.970236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:20.970238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:20.970638 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:20.970916 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:20.971283 (+ 367us) server_negotiation.cc:299] Negotiation successful
0805 22:47:20.971513 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":489,"threads_started":1}
Aug 05 22:47:20 dist-test-slave-7lbf krb5kdc[9313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:20.991443 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:20.977099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33450 (local address 127.4.28.62:37257)
0805 22:47:20.977358 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:20.977364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:20.977447 (+ 83us) server_negotiation.cc:407] Connection header received
0805 22:47:20.977628 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:20.977632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:20.977701 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:20.977785 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:20.979295 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.980051 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.981399 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.981971 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.982083 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:20.986266 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:20.986272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:20.986275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:20.986319 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:20.988760 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:20.989444 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:20.989448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:20.989450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:20.989616 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:20.990014 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:20.990017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:20.990019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:20.990355 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:20.990562 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:20.990827 (+ 265us) server_negotiation.cc:299] Negotiation successful
0805 22:47:20.991133 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260805 22:47:20.992094 9382 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33450
I20260805 22:47:20.993183 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 9458
I20260805 22:47:21.006863 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:20.994162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.4.28.1:35239)
0805 22:47:20.994410 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:20.994417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:20.994571 (+ 154us) server_negotiation.cc:407] Connection header received
0805 22:47:20.994714 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:20.994718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:20.994818 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:20.994907 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:20.996248 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.997291 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.998728 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:20.999158 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:20.999277 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:21.001687 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:21.001693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:21.001696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:21.001739 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:21.004394 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:21.005108 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:21.005113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:21.005116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:21.005263 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:21.005628 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:21.005632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:21.005634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:21.005932 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:21.006140 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:21.006374 (+ 234us) server_negotiation.cc:299] Negotiation successful
0805 22:47:21.006564 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260805 22:47:21.007623 9629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:21.095660 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 9458
I20260805 22:47:21.118970 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 9643
Aug 05 22:47:21 dist-test-slave-7lbf krb5kdc[9313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:47:21.136274 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:21.119933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.4.28.2:35825)
0805 22:47:21.120539 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:21.120548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:21.120579 (+ 31us) server_negotiation.cc:407] Connection header received
0805 22:47:21.120650 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:21.120657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:21.120883 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:21.121038 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:21.122397 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:21.123492 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:21.124933 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:21.125594 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:21.125726 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:21.129924 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:21.129941 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:21.129960 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:21.130004 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:21.132993 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:21.133977 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:21.133986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:21.133989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:21.134198 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:21.134610 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:21.134614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:21.134616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:21.135006 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:21.135270 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:21.135637 (+ 367us) server_negotiation.cc:299] Negotiation successful
0805 22:47:21.135875 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":306,"threads_started":1}
I20260805 22:47:21.137677 9800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:21.220328 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 9643
I20260805 22:47:21.244699 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 9819
Aug 05 22:47:21 dist-test-slave-7lbf krb5kdc[9313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:47:21.263142 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:21.245854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.4.28.3:40969)
0805 22:47:21.246529 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:21.246539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:21.246574 (+ 35us) server_negotiation.cc:407] Connection header received
0805 22:47:21.246642 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:21.246649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:21.246877 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:21.247079 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:21.248509 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:21.249695 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:21.251165 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:21.251784 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:21.251918 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:21.256605 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:21.256622 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:21.256641 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:21.256694 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:21.259814 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:21.260685 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:21.260695 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:21.260697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:21.260864 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:21.261301 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:21.261305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:21.261308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:21.261763 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:21.262134 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:21.262448 (+ 314us) server_negotiation.cc:299] Negotiation successful
0805 22:47:21.262756 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":321,"threads_started":1}
I20260805 22:47:21.264626 10004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:21.349117 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 9819
I20260805 22:47:21.370430 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 9333
I20260805 22:47:21.371837 9422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:21.477238 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 9333
2026-08-05T22:47:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4252 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:21 dist-test-slave-7lbf krb5kdc[10047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:21 dist-test-slave-7lbf krb5kdc[10047](info): set up 2 sockets
Aug 05 22:47:21 dist-test-slave-7lbf krb5kdc[10047](info): commencing operation
krb5kdc: starting...
W20260805 22:47:22.678417 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:22 dist-test-slave-7lbf krb5kdc[10047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:47:22.933945 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44075
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:44075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:23.282863 10061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:23.283218 10061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:23.283355 10061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:23.290972 10061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:23.291126 10061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:23.291195 10061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:23.291275 10061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:23.291343 10061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:23.302626 10061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:44075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:23.304319 10061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:23.305907 10061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:23.318075 10067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:23.318181 10066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:23.318535 10071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:23.319185 10061 server_base.cc:1034] running on GCE node
I20260805 22:47:23.320097 10061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:23.321724 10061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:23.322947 10061 hybrid_clock.cc:648] HybridClock initialized: now 1785970043322885 us; error 60 us; skew 500 ppm
I20260805 22:47:23.323341 10061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:23 dist-test-slave-7lbf krb5kdc[10047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970043, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:23.326855 10061 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:47:23.328306 10061 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:23.328418 10061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:23.328766 10061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:23.332297 10061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90"
format_stamp: "Formatted at 2026-08-05 22:47:23 on dist-test-slave-7lbf"
server_key: "7d04d025df9b12b8c7621fc5f5e7fba7"
server_key_iv: "e822f60aea979f3cd22cab387cf531ab"
server_key_version: "encryptionkey@0"
I20260805 22:47:23.333250 10061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90"
format_stamp: "Formatted at 2026-08-05 22:47:23 on dist-test-slave-7lbf"
server_key: "7d04d025df9b12b8c7621fc5f5e7fba7"
server_key_iv: "e822f60aea979f3cd22cab387cf531ab"
server_key_version: "encryptionkey@0"
I20260805 22:47:23.339273 10061 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260805 22:47:23.343603 10081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:23.345379 10061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260805 22:47:23.345630 10061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90"
format_stamp: "Formatted at 2026-08-05 22:47:23 on dist-test-slave-7lbf"
server_key: "7d04d025df9b12b8c7621fc5f5e7fba7"
server_key_iv: "e822f60aea979f3cd22cab387cf531ab"
server_key_version: "encryptionkey@0"
I20260805 22:47:23.345922 10061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:23.364308 10061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:23.369349 10061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:23.369673 10061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:23.388633 10061 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:44075
I20260805 22:47:23.388659 10157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:44075 every 8 connection(s)
I20260805 22:47:23.389832 10061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:23.394095 10159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:23.399646 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 10061
I20260805 22:47:23.399868 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:23.400524 4208 external_mini_cluster.cc:1428] Setting key 572efa0ff5b13892ed4835efdfcdd18d
I20260805 22:47:23.406157 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90: Bootstrap starting.
I20260805 22:47:23.410045 10159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:23.411414 10159 log.cc:824] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:23.415547 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90: No bootstrap required, opened a new log
Aug 05 22:47:23 dist-test-slave-7lbf krb5kdc[10047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:23.426960 10159 raft_consensus.cc:348] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44075 } }
I20260805 22:47:23.427516 10159 raft_consensus.cc:374] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:23.427654 10159 raft_consensus.cc:729] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46e7b52d8b948d4ab4e3a4e08eeac90, State: Initialized, Role: FOLLOWER
I20260805 22:47:23.428077 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:23.402790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.4.28.62:44075)
0805 22:47:23.403744 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:23.403761 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:23.403813 (+ 52us) server_negotiation.cc:407] Connection header received
0805 22:47:23.405685 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:23.405713 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:23.406172 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:23.406558 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:23.408533 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:23.409966 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:23.411804 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:23.412884 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:23.413084 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:23.418135 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:23.418149 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:23.418172 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:23.418254 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:23.422717 (+ 4463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:23.423571 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:23.423582 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:23.423594 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:23.423863 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:23.424236 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:23.424242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:23.424246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:23.425167 (+ 921us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:23.425493 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:23.425957 (+ 464us) server_negotiation.cc:299] Negotiation successful
0805 22:47:23.426443 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":449,"threads_started":1}
I20260805 22:47:23.428457 10159 consensus_queue.cc:260] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44075 } }
I20260805 22:47:23.428730 10159 raft_consensus.cc:388] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:23.428884 10159 raft_consensus.cc:482] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:23.429065 10159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:23.431674 10159 raft_consensus.cc:504] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44075 } }
I20260805 22:47:23.432210 10159 leader_election.cc:302] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c46e7b52d8b948d4ab4e3a4e08eeac90; no voters:
I20260805 22:47:23.432879 10159 leader_election.cc:288] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:23.433020 10167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:23.433795 10167 raft_consensus.cc:686] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [term 1 LEADER]: Becoming Leader. State: Replica: c46e7b52d8b948d4ab4e3a4e08eeac90, State: Running, Role: LEADER
I20260805 22:47:23.434310 10167 consensus_queue.cc:237] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44075 } }
I20260805 22:47:23.435158 10159 sys_catalog.cc:564] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:23.437040 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44075 } } }
I20260805 22:47:23.437220 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c46e7b52d8b948d4ab4e3a4e08eeac90. Latest consensus state: current_term: 1 leader_uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46e7b52d8b948d4ab4e3a4e08eeac90" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44075 } } }
I20260805 22:47:23.437495 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:23.437492 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:23.440193 10173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:23.450241 10173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:23.462208 10173 catalog_manager.cc:1347] Generated new cluster ID: 7901d3a65e0143b3b6048430118d2679
I20260805 22:47:23.462368 10173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:23.471380 10173 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:23.473475 10173 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:23.481493 10173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90: Generated new TSK 0
I20260805 22:47:23.482721 10173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:47:23.549256 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44075
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:23.913460 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:23.913816 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:23.913932 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:23.921619 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:23.921754 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:23.921883 10195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:23.932417 10195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:23.934128 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:23.935705 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:23.949994 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:23.950064 10206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:23.950024 10204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:23.951236 10195 server_base.cc:1034] running on GCE node
I20260805 22:47:23.951967 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:23.952984 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:23.954203 10195 hybrid_clock.cc:648] HybridClock initialized: now 1785970043954149 us; error 48 us; skew 500 ppm
I20260805 22:47:23.954566 10195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:23 dist-test-slave-7lbf krb5kdc[10047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970043, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:23.957959 10195 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:47:23.959283 10195 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:23.959401 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:23.959744 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:23.963085 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "165b36c8fe7a403b96ea06fb8bf1a057"
format_stamp: "Formatted at 2026-08-05 22:47:23 on dist-test-slave-7lbf"
server_key: "f4d3580e142f6e7a38e19697a9b02399"
server_key_iv: "14cd8cb3f99ec9d73e1c6c3078cdc156"
server_key_version: "encryptionkey@0"
I20260805 22:47:23.964008 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "165b36c8fe7a403b96ea06fb8bf1a057"
format_stamp: "Formatted at 2026-08-05 22:47:23 on dist-test-slave-7lbf"
server_key: "f4d3580e142f6e7a38e19697a9b02399"
server_key_iv: "14cd8cb3f99ec9d73e1c6c3078cdc156"
server_key_version: "encryptionkey@0"
I20260805 22:47:23.969885 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260805 22:47:23.973767 10217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:23.975337 10195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:23.975548 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "165b36c8fe7a403b96ea06fb8bf1a057"
format_stamp: "Formatted at 2026-08-05 22:47:23 on dist-test-slave-7lbf"
server_key: "f4d3580e142f6e7a38e19697a9b02399"
server_key_iv: "14cd8cb3f99ec9d73e1c6c3078cdc156"
server_key_version: "encryptionkey@0"
I20260805 22:47:23.975785 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:23.990090 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:23.994762 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:23.995055 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:23.996088 10195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:23.997579 10195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:23.997677 10195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:23.997762 10195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:23.997817 10195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:24.030961 10195 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:39647
I20260805 22:47:24.031018 10372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:39647 every 8 connection(s)
I20260805 22:47:24.031869 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:24.036087 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 10195
I20260805 22:47:24.036309 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:24.037078 4208 external_mini_cluster.cc:1428] Setting key def972243e05445012cbbcbd839a09b3
Aug 05 22:47:24 dist-test-slave-7lbf krb5kdc[10047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970043, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:24.054809 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:24.034505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:55959 (local address 127.4.28.62:44075)
0805 22:47:24.034978 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:24.034983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:24.036121 (+ 1138us) server_negotiation.cc:407] Connection header received
0805 22:47:24.038538 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:24.038545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:24.038641 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:24.038776 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:24.041031 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:24.041827 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:24.043490 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:24.043919 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:24.044039 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:24.048130 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:24.048137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:24.048140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:24.048185 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:24.051264 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:24.052156 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:24.052163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:24.052167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:24.052313 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:24.052812 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:24.052817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:24.052820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:24.053136 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:24.053281 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:24.054168 (+ 887us) server_negotiation.cc:299] Negotiation successful
0805 22:47:24.054430 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260805 22:47:24.055899 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:24.035035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44075 (local address 127.4.28.1:55959)
0805 22:47:24.035896 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:24.035996 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:24.038178 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:24.039098 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:24.039113 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:24.039958 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:24.040830 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:24.040855 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:24.041986 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:24.041992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:24.043317 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:24.043332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:24.044179 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:24.044188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:24.044441 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:24.045216 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:24.045247 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:24.047819 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:24.051476 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:24.051486 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:24.051504 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:24.052014 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:24.052454 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:24.052458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:24.052460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:24.052690 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:24.053279 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:24.053289 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:24.053745 (+ 456us) client_negotiation.cc:769] Sending connection context
0805 22:47:24.054037 (+ 292us) client_negotiation.cc:240] Negotiation successful
0805 22:47:24.054652 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":386,"threads_started":1}
I20260805 22:47:24.057741 10373 heartbeater.cc:347] Connected to a master server at 127.4.28.62:44075
I20260805 22:47:24.058104 10373 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:24.058827 10373 heartbeater.cc:511] Master 127.4.28.62:44075 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
I20260805 22:47:24.061133 10107 ts_manager.cc:194] Registered new tserver with Master: 165b36c8fe7a403b96ea06fb8bf1a057 (127.4.28.1:39647)
I20260805 22:47:24.062747 10107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:55959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:47:24.096335 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44075
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:24.444765 10381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:24.445116 10381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:24.445220 10381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:24.453419 10381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:24.453609 10381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:24.453747 10381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:24.464586 10381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:24.466187 10381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:24.467602 10381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:24.481585 10387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:24.481572 10395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:24.481638 10388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:24.482436 10381 server_base.cc:1034] running on GCE node
I20260805 22:47:24.483086 10381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:24.484120 10381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:24.485337 10381 hybrid_clock.cc:648] HybridClock initialized: now 1785970044485270 us; error 55 us; skew 500 ppm
I20260805 22:47:24.485754 10381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:24 dist-test-slave-7lbf krb5kdc[10047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970044, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:24.492453 10381 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:47:24.493948 10381 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:24.494091 10381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:24.494453 10381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:24.497819 10381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "8450747e9e334bcfb6c85858090e41c2"
format_stamp: "Formatted at 2026-08-05 22:47:24 on dist-test-slave-7lbf"
server_key: "3693df3aff95096538ad31b3a8510a4f"
server_key_iv: "71a9deb29e9b7fddd19b0ffcdff6e616"
server_key_version: "encryptionkey@0"
I20260805 22:47:24.498785 10381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "8450747e9e334bcfb6c85858090e41c2"
format_stamp: "Formatted at 2026-08-05 22:47:24 on dist-test-slave-7lbf"
server_key: "3693df3aff95096538ad31b3a8510a4f"
server_key_iv: "71a9deb29e9b7fddd19b0ffcdff6e616"
server_key_version: "encryptionkey@0"
I20260805 22:47:24.504993 10381 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260805 22:47:24.509377 10402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:24.511191 10381 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:47:24.511415 10381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "8450747e9e334bcfb6c85858090e41c2"
format_stamp: "Formatted at 2026-08-05 22:47:24 on dist-test-slave-7lbf"
server_key: "3693df3aff95096538ad31b3a8510a4f"
server_key_iv: "71a9deb29e9b7fddd19b0ffcdff6e616"
server_key_version: "encryptionkey@0"
I20260805 22:47:24.511680 10381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:24.527575 10381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:24.534703 10381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:24.535055 10381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:24.536365 10381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:24.538357 10381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:24.538480 10381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:24.538606 10381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:24.538684 10381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:24.580745 10381 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:36125
I20260805 22:47:24.580839 10556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:36125 every 8 connection(s)
I20260805 22:47:24.581736 10381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:24.584301 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 10381
I20260805 22:47:24.584481 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:47:24.584949 4208 external_mini_cluster.cc:1428] Setting key 1cb9f510d5bf234f12871b99827b2065
Aug 05 22:47:24 dist-test-slave-7lbf krb5kdc[10047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970044, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:24.604417 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:24.584248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:34723 (local address 127.4.28.62:44075)
0805 22:47:24.586771 (+ 2523us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:24.586776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:24.586794 (+ 18us) server_negotiation.cc:407] Connection header received
0805 22:47:24.588277 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:24.588282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:24.588366 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:24.588481 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:24.590316 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:24.591312 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:24.593036 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:24.593653 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:24.593787 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:24.597666 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:24.597672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:24.597675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:24.597717 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:24.600509 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:24.601654 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:24.601660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:24.601664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:24.601824 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:24.602401 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:24.602406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:24.602409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:24.602734 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:24.602920 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:24.603922 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0805 22:47:24.604139 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2425,"thread_start_us":2303,"threads_started":1}
I20260805 22:47:24.605922 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:24.584911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44075 (local address 127.4.28.2:34723)
0805 22:47:24.585933 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:24.585988 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:24.587973 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:24.588691 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:24.588700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:24.589253 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:24.590092 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:24.590116 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:24.591498 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:24.591503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:24.592853 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:24.592871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:24.593897 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:24.593903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:24.594118 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:24.594824 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:24.594857 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:24.597359 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:24.600728 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:24.600741 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:24.600765 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:24.601463 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:24.601970 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:24.601975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:24.601978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:24.602249 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:24.602878 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:24.602886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:24.603458 (+ 572us) client_negotiation.cc:769] Sending connection context
0805 22:47:24.603784 (+ 326us) client_negotiation.cc:240] Negotiation successful
0805 22:47:24.604373 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":379,"threads_started":1}
I20260805 22:47:24.607641 10557 heartbeater.cc:347] Connected to a master server at 127.4.28.62:44075
I20260805 22:47:24.608011 10557 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:24.608716 10557 heartbeater.cc:511] Master 127.4.28.62:44075 requested a full tablet report, sending...
I20260805 22:47:24.610258 10107 ts_manager.cc:194] Registered new tserver with Master: 8450747e9e334bcfb6c85858090e41c2 (127.4.28.2:36125)
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
I20260805 22:47:24.611102 10107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:34723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:47:24.649816 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44075
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:25.013295 10570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:25.013646 10570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:25.013792 10570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:25.021389 10570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:25.021559 10570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:25.021682 10570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:25.032563 10570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:25.034036 10570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:25.035496 10570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:25.054768 10579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:25.054701 10582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:25.054798 10580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:25.055951 10570 server_base.cc:1034] running on GCE node
I20260805 22:47:25.056615 10570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:25.057699 10570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:25.058993 10570 hybrid_clock.cc:648] HybridClock initialized: now 1785970045058930 us; error 59 us; skew 500 ppm
I20260805 22:47:25.059363 10570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:25 dist-test-slave-7lbf krb5kdc[10047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970045, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:25.062688 10570 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:47:25.063932 10570 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:25.064039 10570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:25.064338 10570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:25.066782 10373 heartbeater.cc:503] Master 127.4.28.62:44075 was elected leader, sending a full tablet report...
I20260805 22:47:25.067649 10570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "5f81fc98b2f44a24aaf8bccb33f04e46"
format_stamp: "Formatted at 2026-08-05 22:47:25 on dist-test-slave-7lbf"
server_key: "8e824eb09d2e96423445babb4d43558c"
server_key_iv: "b9428c52c6494c7e4360fb03b813ee80"
server_key_version: "encryptionkey@0"
I20260805 22:47:25.068638 10570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "5f81fc98b2f44a24aaf8bccb33f04e46"
format_stamp: "Formatted at 2026-08-05 22:47:25 on dist-test-slave-7lbf"
server_key: "8e824eb09d2e96423445babb4d43558c"
server_key_iv: "b9428c52c6494c7e4360fb03b813ee80"
server_key_version: "encryptionkey@0"
I20260805 22:47:25.074549 10570 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260805 22:47:25.078894 10593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:25.080484 10570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:47:25.080682 10570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "5f81fc98b2f44a24aaf8bccb33f04e46"
format_stamp: "Formatted at 2026-08-05 22:47:25 on dist-test-slave-7lbf"
server_key: "8e824eb09d2e96423445babb4d43558c"
server_key_iv: "b9428c52c6494c7e4360fb03b813ee80"
server_key_version: "encryptionkey@0"
I20260805 22:47:25.080957 10570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:25.101403 10570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:25.105958 10570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:25.106233 10570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:25.107308 10570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:25.108803 10570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:25.108916 10570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:25.109028 10570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:25.109100 10570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:25.142113 10570 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:42277
I20260805 22:47:25.142177 10750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:42277 every 8 connection(s)
I20260805 22:47:25.143050 10570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:25.146683 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 10570
I20260805 22:47:25.146971 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:47:25.147601 4208 external_mini_cluster.cc:1428] Setting key a4a8649ab704bc681e6f909167697fa6
Aug 05 22:47:25 dist-test-slave-7lbf krb5kdc[10047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970045, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:25.165112 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.145743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:41481 (local address 127.4.28.62:44075)
0805 22:47:25.146235 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.146243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.147545 (+ 1302us) server_negotiation.cc:407] Connection header received
0805 22:47:25.150028 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.150033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.150125 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.150248 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:25.152122 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.152925 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.154690 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.155136 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.155233 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.158822 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:25.158828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:25.158831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:25.158884 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:25.161605 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.162615 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.162621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.162625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.162795 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.163319 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.163323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.163326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.163714 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:25.163877 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.164641 (+ 764us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.164834 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":239,"threads_started":1}
I20260805 22:47:25.166173 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.146412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44075 (local address 127.4.28.3:41481)
0805 22:47:25.147308 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:25.147375 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:25.149671 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:25.150465 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:25.150475 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:25.151244 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:25.151940 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.151960 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.153111 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.153116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.154550 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.154565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.155324 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.155328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.155551 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.156216 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:25.156240 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:25.158565 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:25.161785 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:25.161796 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:25.161817 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:25.162475 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:25.162919 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:25.162924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:25.162927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:25.163190 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:25.163863 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:25.163870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:25.164288 (+ 418us) client_negotiation.cc:769] Sending connection context
0805 22:47:25.164540 (+ 252us) client_negotiation.cc:240] Negotiation successful
0805 22:47:25.164973 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":331,"threads_started":1}
I20260805 22:47:25.167615 10751 heartbeater.cc:347] Connected to a master server at 127.4.28.62:44075
I20260805 22:47:25.167939 10751 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:25.168666 10751 heartbeater.cc:511] Master 127.4.28.62:44075 requested a full tablet report, sending...
I20260805 22:47:25.170188 10107 ts_manager.cc:194] Registered new tserver with Master: 5f81fc98b2f44a24aaf8bccb33f04e46 (127.4.28.3:42277)
I20260805 22:47:25.170989 10107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:41481
I20260805 22:47:25.176508 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:47:25.194326 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.181291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.4.28.62:44075)
0805 22:47:25.181571 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.181580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.181988 (+ 408us) server_negotiation.cc:407] Connection header received
0805 22:47:25.182137 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.182141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.182212 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.182297 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:25.184021 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.184810 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.186218 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.186675 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.186776 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.189022 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:25.189027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:25.189030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:25.189069 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:25.191712 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.192513 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.192517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.192519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.192652 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.192959 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.192962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.192963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.193306 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:25.193509 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.193834 (+ 325us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.194032 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 22:47:25.198904 10107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:25.206215 10107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:25.238174 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.221210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:36125 (local address 127.0.0.1:54310)
0805 22:47:25.226111 (+ 4901us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:25.226136 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:25.226322 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:25.229297 (+ 2975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:25.229310 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:25.229377 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:25.229893 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.229910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.232189 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.232196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.235240 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.235256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.236858 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.236865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.237171 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.237249 (+ 78us) client_negotiation.cc:769] Sending connection context
0805 22:47:25.237410 (+ 161us) client_negotiation.cc:240] Negotiation successful
0805 22:47:25.237694 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4729,"mutex_wait_us":25,"thread_start_us":10106,"threads_started":1}
I20260805 22:47:25.238119 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.221295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.4.28.2:36125)
0805 22:47:25.223288 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.223298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.226513 (+ 3215us) server_negotiation.cc:407] Connection header received
0805 22:47:25.226681 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.226693 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.227043 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.227285 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:25.230211 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.231971 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.235439 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.236619 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.236768 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.237008 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.237558 (+ 550us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.237758 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1841,"thread_start_us":1720,"threads_started":1}
I20260805 22:47:25.241250 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.220804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40472 (local address 127.4.28.1:39647)
0805 22:47:25.232153 (+ 11349us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.232161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.232195 (+ 34us) server_negotiation.cc:407] Connection header received
0805 22:47:25.232313 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.232321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.232652 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.233093 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:25.234178 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.235875 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.238524 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.240058 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.240206 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.240456 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.240712 (+ 256us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.240903 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11093,"thread_start_us":11168,"threads_started":1}
I20260805 22:47:25.241245 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.220500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:39647 (local address 127.0.0.1:40472)
0805 22:47:25.222293 (+ 1793us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:25.222388 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:25.222656 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:25.232918 (+ 10262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:25.232924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:25.232955 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:25.233896 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.233915 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.236060 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.236067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.238304 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.238324 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.240300 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.240307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.240486 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.240521 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:47:25.240627 (+ 106us) client_negotiation.cc:240] Negotiation successful
0805 22:47:25.240791 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"mutex_wait_us":58,"thread_start_us":1157,"threads_started":1}
I20260805 22:47:25.241750 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.222671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:42277 (local address 127.0.0.1:60764)
0805 22:47:25.231689 (+ 9018us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:25.231714 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:25.231881 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:25.233573 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:25.233579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:25.233609 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:25.234153 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.234166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.236506 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.236512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.238639 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.238656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.240671 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.240678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.240864 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.240898 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 22:47:25.240974 (+ 76us) client_negotiation.cc:240] Negotiation successful
0805 22:47:25.241119 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8845,"mutex_wait_us":8,"thread_start_us":3078,"threads_started":1}
I20260805 22:47:25.242511 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.223353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.4.28.3:42277)
0805 22:47:25.232863 (+ 9510us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.232871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.232910 (+ 39us) server_negotiation.cc:407] Connection header received
0805 22:47:25.233016 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.233023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.233291 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.234372 (+ 1081us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:25.234708 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.236335 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.238841 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.240075 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.240202 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.241692 (+ 1490us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.241799 (+ 107us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.242156 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9317,"thread_start_us":9075,"threads_started":1}
I20260805 22:47:25.243737 10477 tablet_service.cc:1467] Processing CreateTablet for tablet 746892f65b614807b48eb1c924d06bb3 (DEFAULT_TABLE table=test-table [id=f38cfb7cea644fcbafde9815f140b005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:25.245396 10477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746892f65b614807b48eb1c924d06bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:25.245615 10282 tablet_service.cc:1467] Processing CreateTablet for tablet 746892f65b614807b48eb1c924d06bb3 (DEFAULT_TABLE table=test-table [id=f38cfb7cea644fcbafde9815f140b005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:25.246129 10656 tablet_service.cc:1467] Processing CreateTablet for tablet 746892f65b614807b48eb1c924d06bb3 (DEFAULT_TABLE table=test-table [id=f38cfb7cea644fcbafde9815f140b005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:25.247251 10282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746892f65b614807b48eb1c924d06bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:25.247367 10656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746892f65b614807b48eb1c924d06bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:25.259454 10777 tablet_bootstrap.cc:492] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: Bootstrap starting.
I20260805 22:47:25.262282 10779 tablet_bootstrap.cc:492] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: Bootstrap starting.
I20260805 22:47:25.265429 10777 tablet_bootstrap.cc:654] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:25.266398 10778 tablet_bootstrap.cc:492] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: Bootstrap starting.
I20260805 22:47:25.266470 10779 tablet_bootstrap.cc:654] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:25.267179 10777 log.cc:824] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:25.267717 10779 log.cc:824] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:25.270167 10778 tablet_bootstrap.cc:654] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:25.270721 10777 tablet_bootstrap.cc:492] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: No bootstrap required, opened a new log
I20260805 22:47:25.271016 10777 ts_tablet_manager.cc:1397] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260805 22:47:25.271124 10779 tablet_bootstrap.cc:492] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: No bootstrap required, opened a new log
I20260805 22:47:25.271426 10779 ts_tablet_manager.cc:1397] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260805 22:47:25.271574 10778 log.cc:824] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:25.275128 10778 tablet_bootstrap.cc:492] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: No bootstrap required, opened a new log
I20260805 22:47:25.275445 10778 ts_tablet_manager.cc:1397] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260805 22:47:25.279891 10779 raft_consensus.cc:348] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } }
I20260805 22:47:25.280354 10779 raft_consensus.cc:374] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:25.280459 10779 raft_consensus.cc:729] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165b36c8fe7a403b96ea06fb8bf1a057, State: Initialized, Role: FOLLOWER
I20260805 22:47:25.281124 10779 consensus_queue.cc:260] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } }
I20260805 22:47:25.281549 10777 raft_consensus.cc:348] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } }
I20260805 22:47:25.282035 10777 raft_consensus.cc:374] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:25.282148 10777 raft_consensus.cc:729] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f81fc98b2f44a24aaf8bccb33f04e46, State: Initialized, Role: FOLLOWER
I20260805 22:47:25.282804 10777 consensus_queue.cc:260] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } }
I20260805 22:47:25.284062 10751 heartbeater.cc:503] Master 127.4.28.62:44075 was elected leader, sending a full tablet report...
I20260805 22:47:25.284793 10777 ts_tablet_manager.cc:1428] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260805 22:47:25.286252 10779 ts_tablet_manager.cc:1428] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: Time spent starting tablet: real 0.015s user 0.006s sys 0.007s
I20260805 22:47:25.286410 10778 raft_consensus.cc:348] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } }
I20260805 22:47:25.286825 10778 raft_consensus.cc:374] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:25.286903 10778 raft_consensus.cc:729] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8450747e9e334bcfb6c85858090e41c2, State: Initialized, Role: FOLLOWER
I20260805 22:47:25.287447 10778 consensus_queue.cc:260] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } }
I20260805 22:47:25.288575 10557 heartbeater.cc:503] Master 127.4.28.62:44075 was elected leader, sending a full tablet report...
I20260805 22:47:25.289016 10778 ts_tablet_manager.cc:1428] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260805 22:47:25.333750 10558 tablet.cc:2369] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:47:25.368000 10786 raft_consensus.cc:482] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:25.368330 10786 raft_consensus.cc:504] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } }
I20260805 22:47:25.370251 10786 leader_election.cc:288] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 165b36c8fe7a403b96ea06fb8bf1a057 (127.4.28.1:39647), 8450747e9e334bcfb6c85858090e41c2 (127.4.28.2:36125)
I20260805 22:47:25.376111 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.370751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:36125 (local address 127.4.28.3:54813)
0805 22:47:25.371100 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:25.371121 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:25.371264 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:25.371621 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:25.371626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:25.371653 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:25.371819 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.371828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.372848 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.372853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.374214 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.374228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.375354 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.375361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.375535 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.375561 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 22:47:25.375646 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 22:47:25.375798 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":1}
I20260805 22:47:25.376089 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.370853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:54813 (local address 127.4.28.2:36125)
0805 22:47:25.371170 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.371176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.371196 (+ 20us) server_negotiation.cc:407] Connection header received
0805 22:47:25.371417 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.371423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.371497 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.371573 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:25.371960 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.372696 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.374378 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.375185 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.375316 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.375412 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.375691 (+ 279us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.375826 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260805 22:47:25.377089 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.370784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:57743 (local address 127.4.28.1:39647)
0805 22:47:25.370997 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.371004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.371737 (+ 733us) server_negotiation.cc:407] Connection header received
0805 22:47:25.371910 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.371915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.371976 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.372050 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:25.372599 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.373312 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.375047 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.375823 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.375948 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.376053 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.376778 (+ 725us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.376894 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260805 22:47:25.377205 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.370797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:39647 (local address 127.4.28.3:57743)
0805 22:47:25.371622 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:25.371637 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:25.371775 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:25.372083 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:25.372087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:25.372111 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:25.372429 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.372439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.373471 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.373475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.374879 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:25.374892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.376435 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.376446 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:25.376609 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.376635 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 22:47:25.376730 (+ 95us) client_negotiation.cc:240] Negotiation successful
0805 22:47:25.376872 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":445,"threads_started":1}
I20260805 22:47:25.377326 10504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746892f65b614807b48eb1c924d06bb3" candidate_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8450747e9e334bcfb6c85858090e41c2" is_pre_election: true
I20260805 22:47:25.378063 10504 raft_consensus.cc:2454] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f81fc98b2f44a24aaf8bccb33f04e46 in term 0.
I20260805 22:47:25.378389 10317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746892f65b614807b48eb1c924d06bb3" candidate_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" is_pre_election: true
I20260805 22:47:25.378840 10597 leader_election.cc:302] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f81fc98b2f44a24aaf8bccb33f04e46, 8450747e9e334bcfb6c85858090e41c2; no voters:
I20260805 22:47:25.378962 10317 raft_consensus.cc:2454] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f81fc98b2f44a24aaf8bccb33f04e46 in term 0.
I20260805 22:47:25.379309 10786 raft_consensus.cc:2781] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:25.379491 10786 raft_consensus.cc:482] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:25.379606 10786 raft_consensus.cc:3037] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:25.382452 10786 raft_consensus.cc:504] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } }
I20260805 22:47:25.383165 10786 leader_election.cc:288] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [CANDIDATE]: Term 1 election: Requested vote from peers 165b36c8fe7a403b96ea06fb8bf1a057 (127.4.28.1:39647), 8450747e9e334bcfb6c85858090e41c2 (127.4.28.2:36125)
I20260805 22:47:25.383772 10504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746892f65b614807b48eb1c924d06bb3" candidate_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8450747e9e334bcfb6c85858090e41c2"
I20260805 22:47:25.383772 10317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746892f65b614807b48eb1c924d06bb3" candidate_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165b36c8fe7a403b96ea06fb8bf1a057"
I20260805 22:47:25.384018 10317 raft_consensus.cc:3037] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:25.384017 10504 raft_consensus.cc:3037] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:25.387461 10317 raft_consensus.cc:2454] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f81fc98b2f44a24aaf8bccb33f04e46 in term 1.
I20260805 22:47:25.387461 10504 raft_consensus.cc:2454] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f81fc98b2f44a24aaf8bccb33f04e46 in term 1.
I20260805 22:47:25.388067 10597 leader_election.cc:302] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f81fc98b2f44a24aaf8bccb33f04e46, 8450747e9e334bcfb6c85858090e41c2; no voters:
I20260805 22:47:25.388448 10786 raft_consensus.cc:2781] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:25.388954 10786 raft_consensus.cc:686] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 1 LEADER]: Becoming Leader. State: Replica: 5f81fc98b2f44a24aaf8bccb33f04e46, State: Running, Role: LEADER
I20260805 22:47:25.389492 10786 consensus_queue.cc:237] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } }
W20260805 22:47:25.394570 10752 tablet.cc:2369] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:47:25.395831 10103 catalog_manager.cc:5495] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f81fc98b2f44a24aaf8bccb33f04e46 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f81fc98b2f44a24aaf8bccb33f04e46" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42277 } health_report { overall_health: HEALTHY } } }
I20260805 22:47:25.432407 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.426631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.4.28.3:42277)
0805 22:47:25.426877 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.426884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.426977 (+ 93us) server_negotiation.cc:407] Connection header received
0805 22:47:25.427225 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.427229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.427326 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.427471 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:25.427885 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.428639 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.430081 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.430497 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.430633 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.431200 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:25.431770 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:25.431915 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.432043 (+ 128us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.432139 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260805 22:47:25.441618 10317 raft_consensus.cc:1261] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [term 1 FOLLOWER]: Refusing update from remote peer 5f81fc98b2f44a24aaf8bccb33f04e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:25.441624 10504 raft_consensus.cc:1261] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [term 1 FOLLOWER]: Refusing update from remote peer 5f81fc98b2f44a24aaf8bccb33f04e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:25.442801 10789 consensus_queue.cc:1035] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8450747e9e334bcfb6c85858090e41c2" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:25.443220 10786 consensus_queue.cc:1035] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "165b36c8fe7a403b96ea06fb8bf1a057" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:25.476037 10107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:25.476320 10107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:25.480564 10107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c46e7b52d8b948d4ab4e3a4e08eeac90: Sending DeleteTablet for 3 replicas of tablet 746892f65b614807b48eb1c924d06bb3
I20260805 22:47:25.481993 10477 tablet_service.cc:1514] Processing DeleteTablet for tablet 746892f65b614807b48eb1c924d06bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:25 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:54310
I20260805 22:47:25.482059 10282 tablet_service.cc:1514] Processing DeleteTablet for tablet 746892f65b614807b48eb1c924d06bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:25 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:40472
I20260805 22:47:25.482796 10656 tablet_service.cc:1514] Processing DeleteTablet for tablet 746892f65b614807b48eb1c924d06bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:25 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:60764
I20260805 22:47:25.483071 10810 tablet_replica.cc:330] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: stopping tablet replica
I20260805 22:47:25.483630 10810 raft_consensus.cc:2229] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:25.483745 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 10195
I20260805 22:47:25.486930 10811 tablet_replica.cc:330] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: stopping tablet replica
I20260805 22:47:25.487334 10811 raft_consensus.cc:2229] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:25.487802 10811 raft_consensus.cc:2258] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:25.489720 10811 ts_tablet_manager.cc:1905] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:25.491405 10810 raft_consensus.cc:2258] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:25.494040 10810 ts_tablet_manager.cc:1905] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:25.496315 10811 ts_tablet_manager.cc:1918] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:25.496541 10811 log.cc:1196] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/746892f65b614807b48eb1c924d06bb3
I20260805 22:47:25.497119 10811 ts_tablet_manager.cc:1939] T 746892f65b614807b48eb1c924d06bb3 P 165b36c8fe7a403b96ea06fb8bf1a057: Deleting consensus metadata
I20260805 22:47:25.499347 10083 catalog_manager.cc:4841] TS 165b36c8fe7a403b96ea06fb8bf1a057 (127.4.28.1:39647): tablet 746892f65b614807b48eb1c924d06bb3 (table test-table [id=f38cfb7cea644fcbafde9815f140b005]) successfully deleted
I20260805 22:47:25.501199 10810 ts_tablet_manager.cc:1918] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:25.501377 10810 log.cc:1196] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/746892f65b614807b48eb1c924d06bb3
I20260805 22:47:25.501972 10810 ts_tablet_manager.cc:1939] T 746892f65b614807b48eb1c924d06bb3 P 8450747e9e334bcfb6c85858090e41c2: Deleting consensus metadata
I20260805 22:47:25.503911 10084 catalog_manager.cc:4841] TS 8450747e9e334bcfb6c85858090e41c2 (127.4.28.2:36125): tablet 746892f65b614807b48eb1c924d06bb3 (table test-table [id=f38cfb7cea644fcbafde9815f140b005]) successfully deleted
I20260805 22:47:25.506145 10813 tablet_replica.cc:330] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: stopping tablet replica
Aug 05 22:47:25 dist-test-slave-7lbf krb5kdc[10047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:47:25.506662 10813 raft_consensus.cc:2229] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:25.507332 10813 raft_consensus.cc:2258] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:25.509658 10813 ts_tablet_manager.cc:1905] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:25.515569 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.485653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.4.28.1:39647)
0805 22:47:25.485874 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.485881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.495488 (+ 9607us) server_negotiation.cc:407] Connection header received
0805 22:47:25.496056 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.496064 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.496181 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.496267 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:25.498427 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.499606 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.501653 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.502278 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.502426 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.507786 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:25.507806 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:25.507828 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:25.507878 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:25.511894 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.512957 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.512969 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.512972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.513189 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.513717 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.513723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.513726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.514253 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:25.514590 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.514916 (+ 326us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.515175 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 22:47:25.516688 10367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:25.517272 10813 ts_tablet_manager.cc:1918] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:25.517467 10813 log.cc:1196] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/746892f65b614807b48eb1c924d06bb3
I20260805 22:47:25.518087 10813 ts_tablet_manager.cc:1939] T 746892f65b614807b48eb1c924d06bb3 P 5f81fc98b2f44a24aaf8bccb33f04e46: Deleting consensus metadata
I20260805 22:47:25.520159 10083 catalog_manager.cc:4841] TS 5f81fc98b2f44a24aaf8bccb33f04e46 (127.4.28.3:42277): tablet 746892f65b614807b48eb1c924d06bb3 (table test-table [id=f38cfb7cea644fcbafde9815f140b005]) successfully deleted
I20260805 22:47:25.617781 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 10195
I20260805 22:47:25.643510 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 10381
Aug 05 22:47:25 dist-test-slave-7lbf krb5kdc[10047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:47:25.660261 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.644316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.4.28.2:36125)
0805 22:47:25.644527 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.644534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.644928 (+ 394us) server_negotiation.cc:407] Connection header received
0805 22:47:25.645104 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.645110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.645190 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.645277 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:25.646614 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.647414 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.648689 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.649109 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.649228 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.653866 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:25.653883 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:25.653903 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:25.653948 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:25.657028 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.657982 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.657991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.657994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.658145 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.658582 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.658586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.658588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.659029 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:25.659343 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.659686 (+ 343us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.659937 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:47:25.660950 10552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:25.751343 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 10381
I20260805 22:47:25.777012 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 10570
Aug 05 22:47:25 dist-test-slave-7lbf krb5kdc[10047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:47:25.795085 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:25.777862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.4.28.3:42277)
0805 22:47:25.778062 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:25.778069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:25.778163 (+ 94us) server_negotiation.cc:407] Connection header received
0805 22:47:25.778376 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:25.778382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:25.778480 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:25.778575 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:25.779955 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.780841 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.782359 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:25.782853 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:25.782987 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:25.788173 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:25.788194 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:25.788214 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:25.788262 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:25.791650 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.792550 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.792559 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.792562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.792750 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:25.793217 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:25.793222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:25.793224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:25.793761 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:25.794096 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:25.794470 (+ 374us) server_negotiation.cc:299] Negotiation successful
0805 22:47:25.794712 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260805 22:47:25.795845 10744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:25.920050 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 10570
I20260805 22:47:25.945215 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 10061
I20260805 22:47:25.946602 10153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:26.058722 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 10061
2026-08-05T22:47:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4585 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-05T22:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:26Z Disabled control of system clock
I20260805 22:47:26.146699 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45821
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260805 22:47:26.498711 10828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:26.499053 10828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:26.499142 10828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:26.506683 10828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:26.506822 10828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:26.506860 10828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:26.506917 10828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:26.506961 10828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:26.516568 10828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:26.517997 10828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:26.519915 10828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:26.532426 10835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:26.532454 10839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:26.532394 10841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:26.533320 10828 server_base.cc:1034] running on GCE node
I20260805 22:47:26.534272 10828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:26.535892 10828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:26.537099 10828 hybrid_clock.cc:648] HybridClock initialized: now 1785970046537048 us; error 63 us; skew 500 ppm
I20260805 22:47:26.537469 10828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:47:26.539893 10828 webserver.cc:460] Webserver started at http://127.4.28.62:36259/ using document root <none> and password file <none>
I20260805 22:47:26.540669 10828 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:26.540771 10828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:26.541152 10828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:26.544353 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "3c4425d814a343f49a3b98a19ff3290c"
format_stamp: "Formatted at 2026-08-05 22:47:26 on dist-test-slave-7lbf"
server_key: "43fe9bbfbfb13e5b7422ffb906e135e2"
server_key_iv: "3a387617bc95c17785ba1b3ba44384ee"
server_key_version: "encryptionkey@0"
I20260805 22:47:26.545240 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "3c4425d814a343f49a3b98a19ff3290c"
format_stamp: "Formatted at 2026-08-05 22:47:26 on dist-test-slave-7lbf"
server_key: "43fe9bbfbfb13e5b7422ffb906e135e2"
server_key_iv: "3a387617bc95c17785ba1b3ba44384ee"
server_key_version: "encryptionkey@0"
I20260805 22:47:26.551112 10828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260805 22:47:26.555056 10850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:26.556635 10828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:26.556836 10828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "3c4425d814a343f49a3b98a19ff3290c"
format_stamp: "Formatted at 2026-08-05 22:47:26 on dist-test-slave-7lbf"
server_key: "43fe9bbfbfb13e5b7422ffb906e135e2"
server_key_iv: "3a387617bc95c17785ba1b3ba44384ee"
server_key_version: "encryptionkey@0"
I20260805 22:47:26.557078 10828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:26.563267 10828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:26.564133 10828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:26.564429 10828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:26.583039 10828 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:45821
I20260805 22:47:26.583078 10924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:45821 every 8 connection(s)
I20260805 22:47:26.584471 10828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:26.588712 10926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:26.592512 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 10828
I20260805 22:47:26.592720 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:26.593361 4208 external_mini_cluster.cc:1428] Setting key 69d4b195959b14715e08d5932ccb1fc8
I20260805 22:47:26.600814 10926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c: Bootstrap starting.
I20260805 22:47:26.603153 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:26.595608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.4.28.62:45821)
0805 22:47:26.596732 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:26.596745 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:26.596791 (+ 46us) server_negotiation.cc:407] Connection header received
0805 22:47:26.599117 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:26.599149 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:26.599688 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:26.600113 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:26.601312 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:26.601320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:26.601340 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:26.601356 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:26.601426 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:26.601444 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:26.601728 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:26.601897 (+ 169us) server_negotiation.cc:299] Negotiation successful
0805 22:47:26.602174 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":382,"threads_started":1}
I20260805 22:47:26.604717 10926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:26.605880 10926 log.cc:824] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:26.608889 10926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c: No bootstrap required, opened a new log
I20260805 22:47:26.616808 10926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4425d814a343f49a3b98a19ff3290c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45821 } }
I20260805 22:47:26.617223 10926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:26.617311 10926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4425d814a343f49a3b98a19ff3290c, State: Initialized, Role: FOLLOWER
I20260805 22:47:26.618009 10926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4425d814a343f49a3b98a19ff3290c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45821 } }
I20260805 22:47:26.618273 10926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:26.618410 10926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:26.618593 10926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:26.620882 10926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4425d814a343f49a3b98a19ff3290c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45821 } }
I20260805 22:47:26.621444 10926 leader_election.cc:302] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c4425d814a343f49a3b98a19ff3290c; no voters:
I20260805 22:47:26.622107 10926 leader_election.cc:288] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:26.622257 10933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:26.622999 10933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [term 1 LEADER]: Becoming Leader. State: Replica: 3c4425d814a343f49a3b98a19ff3290c, State: Running, Role: LEADER
I20260805 22:47:26.623580 10933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4425d814a343f49a3b98a19ff3290c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45821 } }
I20260805 22:47:26.624296 10926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:26.626539 10937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c4425d814a343f49a3b98a19ff3290c. Latest consensus state: current_term: 1 leader_uuid: "3c4425d814a343f49a3b98a19ff3290c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4425d814a343f49a3b98a19ff3290c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45821 } } }
I20260805 22:47:26.626979 10937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:26.627509 10935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c4425d814a343f49a3b98a19ff3290c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4425d814a343f49a3b98a19ff3290c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45821 } } }
I20260805 22:47:26.627885 10935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:26.629673 10946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:26.635469 10946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:26.646186 10946 catalog_manager.cc:1347] Generated new cluster ID: 9739cb4bef214a5d97c0ac56ba2c1152
I20260805 22:47:26.646320 10946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:26.663013 10946 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:26.663532 10946 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:26.677065 10946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c: Generated new TSK 0
I20260805 22:47:26.678313 10946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:47:26.693168 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45821
--builtin_ntp_servers=127.4.28.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260805 22:47:27.060258 10960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:27.060590 10960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:27.060712 10960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:27.068578 10960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:27.068696 10960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:27.068890 10960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:27.078585 10960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:27.080022 10960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:27.082101 10960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:27.096462 10966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:27.096491 10968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:27.096422 10970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:27.097468 10960 server_base.cc:1034] running on GCE node
I20260805 22:47:27.098215 10960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:27.099329 10960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:27.100564 10960 hybrid_clock.cc:648] HybridClock initialized: now 1785970047100504 us; error 59 us; skew 500 ppm
I20260805 22:47:27.100907 10960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:47:27.103641 10960 webserver.cc:460] Webserver started at http://127.4.28.1:35451/ using document root <none> and password file <none>
I20260805 22:47:27.104460 10960 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:27.104594 10960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:27.104943 10960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:27.108204 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "cbc84e50f73a40dfa3f145e7edf8764f"
format_stamp: "Formatted at 2026-08-05 22:47:27 on dist-test-slave-7lbf"
server_key: "226fc19949d3a6e45fb69d828417680d"
server_key_iv: "104edd77343060ff77d4018b10ffcd08"
server_key_version: "encryptionkey@0"
I20260805 22:47:27.109112 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "cbc84e50f73a40dfa3f145e7edf8764f"
format_stamp: "Formatted at 2026-08-05 22:47:27 on dist-test-slave-7lbf"
server_key: "226fc19949d3a6e45fb69d828417680d"
server_key_iv: "104edd77343060ff77d4018b10ffcd08"
server_key_version: "encryptionkey@0"
I20260805 22:47:27.115161 10960 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260805 22:47:27.120210 10978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:27.121961 10960 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260805 22:47:27.122169 10960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "cbc84e50f73a40dfa3f145e7edf8764f"
format_stamp: "Formatted at 2026-08-05 22:47:27 on dist-test-slave-7lbf"
server_key: "226fc19949d3a6e45fb69d828417680d"
server_key_iv: "104edd77343060ff77d4018b10ffcd08"
server_key_version: "encryptionkey@0"
I20260805 22:47:27.122460 10960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:27.129308 10960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:27.130458 10960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:27.130759 10960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:27.131793 10960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:27.133402 10960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:27.133507 10960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:27.133646 10960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:27.133719 10960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:27.166307 10960 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:42129
I20260805 22:47:27.166365 11150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:42129 every 8 connection(s)
I20260805 22:47:27.167829 10960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:27.169907 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 10960
I20260805 22:47:27.170152 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:27.170863 4208 external_mini_cluster.cc:1428] Setting key 0845ebb363f98cce759cb7a8ae3d4227
I20260805 22:47:27.173828 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45821
--builtin_ntp_servers=127.4.28.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260805 22:47:27.178668 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:27.171836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45821 (local address 127.4.28.1:43553)
0805 22:47:27.173061 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:27.173117 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:27.174802 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:27.175540 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:27.175549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:27.175652 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:27.176383 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:47:27.176386 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:27.176432 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:47:27.176534 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:27.176997 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:27.177010 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:27.177019 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 22:47:27.177362 (+ 343us) client_negotiation.cc:240] Negotiation successful
0805 22:47:27.177657 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":355,"threads_started":1}
I20260805 22:47:27.179422 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:27.171137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:43553 (local address 127.4.28.62:45821)
0805 22:47:27.171681 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:27.171688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:27.173228 (+ 1540us) server_negotiation.cc:407] Connection header received
0805 22:47:27.175094 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:27.175100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:27.175172 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:27.175270 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:27.176754 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:27.176764 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:27.176768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:27.176789 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:27.176833 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:27.176847 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:27.177017 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:27.179072 (+ 2055us) server_negotiation.cc:299] Negotiation successful
0805 22:47:27.179147 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":252,"threads_started":1}
I20260805 22:47:27.180794 11151 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45821
I20260805 22:47:27.181186 11151 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:27.182044 11151 heartbeater.cc:511] Master 127.4.28.62:45821 requested a full tablet report, sending...
I20260805 22:47:27.184616 10875 ts_manager.cc:194] Registered new tserver with Master: cbc84e50f73a40dfa3f145e7edf8764f (127.4.28.1:42129)
W20260805 22:47:27.524128 11157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:27.524509 11157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:27.524655 11157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:27.532686 11157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:27.532823 11157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:27.532984 11157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:27.542634 11157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:27.544039 11157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:27.546021 11157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:27.560151 11170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:27.560305 11167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:27.560086 11168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:27.561131 11157 server_base.cc:1034] running on GCE node
I20260805 22:47:27.561857 11157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:27.562903 11157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:27.564131 11157 hybrid_clock.cc:648] HybridClock initialized: now 1785970047564076 us; error 49 us; skew 500 ppm
I20260805 22:47:27.564469 11157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:47:27.567047 11157 webserver.cc:460] Webserver started at http://127.4.28.2:36003/ using document root <none> and password file <none>
I20260805 22:47:27.567782 11157 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:27.567894 11157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:27.568240 11157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:27.571524 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "165eccd0e67340b9b3c79b74ff10611e"
format_stamp: "Formatted at 2026-08-05 22:47:27 on dist-test-slave-7lbf"
server_key: "52d62cb3f32877cff3745a0778333346"
server_key_iv: "ddf060ae8a499032b5a6b8a4feb3bf06"
server_key_version: "encryptionkey@0"
I20260805 22:47:27.572363 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "165eccd0e67340b9b3c79b74ff10611e"
format_stamp: "Formatted at 2026-08-05 22:47:27 on dist-test-slave-7lbf"
server_key: "52d62cb3f32877cff3745a0778333346"
server_key_iv: "ddf060ae8a499032b5a6b8a4feb3bf06"
server_key_version: "encryptionkey@0"
I20260805 22:47:27.578145 11157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260805 22:47:27.582113 11180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:27.583705 11157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:47:27.583869 11157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "165eccd0e67340b9b3c79b74ff10611e"
format_stamp: "Formatted at 2026-08-05 22:47:27 on dist-test-slave-7lbf"
server_key: "52d62cb3f32877cff3745a0778333346"
server_key_iv: "ddf060ae8a499032b5a6b8a4feb3bf06"
server_key_version: "encryptionkey@0"
I20260805 22:47:27.584117 11157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:27.590189 11157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:27.591094 11157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:27.591363 11157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:27.592376 11157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:27.594004 11157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:27.594099 11157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:27.594214 11157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:27.594283 11157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:27.626274 11157 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:42529
I20260805 22:47:27.626307 11342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:42529 every 8 connection(s)
I20260805 22:47:27.627880 11157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:27.630663 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 11157
I20260805 22:47:27.630897 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:47:27.631417 4208 external_mini_cluster.cc:1428] Setting key 78fc0699d9025de5d95e702d5219196c
I20260805 22:47:27.634435 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45821
--builtin_ntp_servers=127.4.28.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260805 22:47:27.636925 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:27.630708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:48317 (local address 127.4.28.62:45821)
0805 22:47:27.630934 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:27.630940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:27.632617 (+ 1677us) server_negotiation.cc:407] Connection header received
0805 22:47:27.634489 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:27.634497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:27.634588 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:27.634704 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:27.636201 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:27.636208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:27.636212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:27.636235 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:27.636278 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:27.636293 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:27.636417 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:27.636582 (+ 165us) server_negotiation.cc:299] Negotiation successful
0805 22:47:27.636654 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 22:47:27.638070 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:27.631317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45821 (local address 127.4.28.2:48317)
0805 22:47:27.632441 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:27.632491 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:27.634178 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:27.634954 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:27.634964 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:27.635079 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:27.635807 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:47:27.635810 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:27.635856 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:47:27.635959 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:27.636391 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:27.636398 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:27.636405 (+ 7us) client_negotiation.cc:769] Sending connection context
0805 22:47:27.636640 (+ 235us) client_negotiation.cc:240] Negotiation successful
0805 22:47:27.636879 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":345,"threads_started":1}
I20260805 22:47:27.639626 11343 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45821
I20260805 22:47:27.639953 11343 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:27.640612 11343 heartbeater.cc:511] Master 127.4.28.62:45821 requested a full tablet report, sending...
I20260805 22:47:27.642305 10875 ts_manager.cc:194] Registered new tserver with Master: 165eccd0e67340b9b3c79b74ff10611e (127.4.28.2:42529)
W20260805 22:47:27.995949 11346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:27.996279 11346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:27.996408 11346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:28.003858 11346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:28.003998 11346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:28.004140 11346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:28.013742 11346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:28.015158 11346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:28.017110 11346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:28.031504 11352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:28.031541 11355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:28.031489 11357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:28.032790 11346 server_base.cc:1034] running on GCE node
I20260805 22:47:28.033547 11346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:28.034577 11346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:28.035810 11346 hybrid_clock.cc:648] HybridClock initialized: now 1785970048035750 us; error 43 us; skew 500 ppm
I20260805 22:47:28.036196 11346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:47:28.038626 11346 webserver.cc:460] Webserver started at http://127.4.28.3:41007/ using document root <none> and password file <none>
I20260805 22:47:28.039373 11346 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:28.039472 11346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:28.039824 11346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:28.042992 11346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "fef2508dd109430caa488ea8de6faf64"
format_stamp: "Formatted at 2026-08-05 22:47:28 on dist-test-slave-7lbf"
server_key: "a86eb43288f9b26e8dd6420fbaebac37"
server_key_iv: "a568f1e3b68bb6c62b7d8a3fbc7f9595"
server_key_version: "encryptionkey@0"
I20260805 22:47:28.043875 11346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "fef2508dd109430caa488ea8de6faf64"
format_stamp: "Formatted at 2026-08-05 22:47:28 on dist-test-slave-7lbf"
server_key: "a86eb43288f9b26e8dd6420fbaebac37"
server_key_iv: "a568f1e3b68bb6c62b7d8a3fbc7f9595"
server_key_version: "encryptionkey@0"
I20260805 22:47:28.049602 11346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260805 22:47:28.054572 11368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:28.056154 11346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:47:28.056355 11346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "fef2508dd109430caa488ea8de6faf64"
format_stamp: "Formatted at 2026-08-05 22:47:28 on dist-test-slave-7lbf"
server_key: "a86eb43288f9b26e8dd6420fbaebac37"
server_key_iv: "a568f1e3b68bb6c62b7d8a3fbc7f9595"
server_key_version: "encryptionkey@0"
I20260805 22:47:28.056638 11346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:28.063083 11346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:28.064056 11346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:28.064378 11346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:28.065434 11346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:28.067193 11346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:28.067313 11346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:28.067451 11346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:28.067534 11346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:28.102240 11346 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:34621
I20260805 22:47:28.102290 11524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:34621 every 8 connection(s)
I20260805 22:47:28.103768 11346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:28.111441 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 11346
I20260805 22:47:28.111653 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:47:28.112208 4208 external_mini_cluster.cc:1428] Setting key 82449e18a2d39844a7fc682590c1861d
I20260805 22:47:28.112531 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.106584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:54051 (local address 127.4.28.62:45821)
0805 22:47:28.106798 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.106803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.108298 (+ 1495us) server_negotiation.cc:407] Connection header received
0805 22:47:28.110122 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.110127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.110190 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.110284 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.111773 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.111778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.111782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.111801 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.111834 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.111848 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.111951 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.112193 (+ 242us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.112252 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260805 22:47:28.113785 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.107125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45821 (local address 127.4.28.3:54051)
0805 22:47:28.108133 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.108180 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:28.109830 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:28.110497 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:28.110505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:28.110622 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:28.111361 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:47:28.111364 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:28.111425 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:47:28.111552 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:28.111954 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:28.111967 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:28.111975 (+ 8us) client_negotiation.cc:769] Sending connection context
0805 22:47:28.112278 (+ 303us) client_negotiation.cc:240] Negotiation successful
0805 22:47:28.112589 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":298,"threads_started":1}
I20260805 22:47:28.115337 11526 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45821
I20260805 22:47:28.115710 11526 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:28.116371 11526 heartbeater.cc:511] Master 127.4.28.62:45821 requested a full tablet report, sending...
I20260805 22:47:28.118016 10875 ts_manager.cc:194] Registered new tserver with Master: fef2508dd109430caa488ea8de6faf64 (127.4.28.3:34621)
I20260805 22:47:28.128209 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:47:28.135597 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.133152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.4.28.62:45821)
0805 22:47:28.133331 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.133336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.133882 (+ 546us) server_negotiation.cc:407] Connection header received
0805 22:47:28.134072 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.134076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.134139 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.134233 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.135104 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.135109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.135112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.135130 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.135166 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.135178 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.135288 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.135322 (+ 34us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.135372 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 22:47:28.139706 10875 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:28.146669 10875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:28.166097 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.159804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42529 (local address 127.0.0.1:36078)
0805 22:47:28.161961 (+ 2157us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.162037 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:28.162264 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:28.163379 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:28.163387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:28.163482 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:28.164635 (+ 1153us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:47:28.164638 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:28.164671 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:47:28.164765 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:28.165310 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:28.165322 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:28.165330 (+ 8us) client_negotiation.cc:769] Sending connection context
0805 22:47:28.165500 (+ 170us) client_negotiation.cc:240] Negotiation successful
0805 22:47:28.165675 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1839,"thread_start_us":3001,"threads_started":1}
I20260805 22:47:28.166785 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.159999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.4.28.2:42529)
0805 22:47:28.162755 (+ 2756us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.162765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.162798 (+ 33us) server_negotiation.cc:407] Connection header received
0805 22:47:28.162874 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.162884 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.163164 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.164002 (+ 838us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.164990 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.164999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.165014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.165033 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.165092 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.165119 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.165253 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.166351 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.166476 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2603,"thread_start_us":2489,"threads_started":1}
I20260805 22:47:28.169490 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.160088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42129 (local address 127.0.0.1:57892)
0805 22:47:28.163079 (+ 2991us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.163102 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:28.163198 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:28.167126 (+ 3928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:28.167131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:28.167154 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:28.168185 (+ 1031us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:47:28.168188 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:28.168206 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:47:28.168265 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:28.168840 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:28.168845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:28.168849 (+ 4us) client_negotiation.cc:769] Sending connection context
0805 22:47:28.168985 (+ 136us) client_negotiation.cc:240] Negotiation successful
0805 22:47:28.169041 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2861,"thread_start_us":1349,"threads_started":1}
I20260805 22:47:28.170367 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.160723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.4.28.1:42129)
0805 22:47:28.165955 (+ 5232us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.165966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.165999 (+ 33us) server_negotiation.cc:407] Connection header received
0805 22:47:28.166086 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.166094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.166341 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.166495 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.168454 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.168463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.168478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.168499 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.168574 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.168601 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.168780 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.169768 (+ 988us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.169882 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4968,"thread_start_us":7891,"threads_started":1}
I20260805 22:47:28.173095 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.162417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:34621 (local address 127.0.0.1:38592)
0805 22:47:28.165261 (+ 2844us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.165284 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:28.165364 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:28.169301 (+ 3937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:28.169310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:28.169333 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:28.170051 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:47:28.170054 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:28.170072 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:47:28.170118 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:28.172448 (+ 2330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:28.172453 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:28.172458 (+ 5us) client_negotiation.cc:769] Sending connection context
0805 22:47:28.172592 (+ 134us) client_negotiation.cc:240] Negotiation successful
0805 22:47:28.172649 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2731,"thread_start_us":2601,"threads_started":1}
I20260805 22:47:28.173054 11249 tablet_service.cc:1467] Processing CreateTablet for tablet fa6a2308ee384bc4b0c1daef92324314 (DEFAULT_TABLE table=test-table [id=c88607fd699444f48dba12990f9bc730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:28.174201 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.163003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.4.28.3:34621)
0805 22:47:28.168744 (+ 5741us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.168752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.168787 (+ 35us) server_negotiation.cc:407] Connection header received
0805 22:47:28.168854 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.168862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.169110 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.169797 (+ 687us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.171226 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.171235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.171248 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.171266 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.172285 (+ 1019us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.172316 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.173610 (+ 1294us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.173689 (+ 79us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.173829 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5563,"thread_start_us":5390,"threads_started":1}
I20260805 22:47:28.174678 11249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6a2308ee384bc4b0c1daef92324314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:28.174942 11049 tablet_service.cc:1467] Processing CreateTablet for tablet fa6a2308ee384bc4b0c1daef92324314 (DEFAULT_TABLE table=test-table [id=c88607fd699444f48dba12990f9bc730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:28.176257 11049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6a2308ee384bc4b0c1daef92324314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:28.178607 11438 tablet_service.cc:1467] Processing CreateTablet for tablet fa6a2308ee384bc4b0c1daef92324314 (DEFAULT_TABLE table=test-table [id=c88607fd699444f48dba12990f9bc730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:28.180154 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6a2308ee384bc4b0c1daef92324314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:28.187909 11151 heartbeater.cc:503] Master 127.4.28.62:45821 was elected leader, sending a full tablet report...
I20260805 22:47:28.189234 11560 tablet_bootstrap.cc:492] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: Bootstrap starting.
I20260805 22:47:28.191322 11562 tablet_bootstrap.cc:492] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f: Bootstrap starting.
I20260805 22:47:28.193061 11560 tablet_bootstrap.cc:654] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:28.194526 11560 log.cc:824] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:28.195230 11563 tablet_bootstrap.cc:492] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: Bootstrap starting.
I20260805 22:47:28.196141 11562 tablet_bootstrap.cc:654] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:28.197492 11562 log.cc:824] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:28.198552 11560 tablet_bootstrap.cc:492] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: No bootstrap required, opened a new log
I20260805 22:47:28.198534 11563 tablet_bootstrap.cc:654] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:28.198828 11560 ts_tablet_manager.cc:1397] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260805 22:47:28.199618 11563 log.cc:824] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:28.200963 11562 tablet_bootstrap.cc:492] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f: No bootstrap required, opened a new log
I20260805 22:47:28.201290 11562 ts_tablet_manager.cc:1397] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260805 22:47:28.202572 11563 tablet_bootstrap.cc:492] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: No bootstrap required, opened a new log
I20260805 22:47:28.203233 11563 ts_tablet_manager.cc:1397] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260805 22:47:28.209925 11560 raft_consensus.cc:348] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } }
I20260805 22:47:28.210520 11560 raft_consensus.cc:374] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:28.210659 11560 raft_consensus.cc:729] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165eccd0e67340b9b3c79b74ff10611e, State: Initialized, Role: FOLLOWER
I20260805 22:47:28.210410 11563 raft_consensus.cc:348] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } }
I20260805 22:47:28.210891 11563 raft_consensus.cc:374] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:28.210984 11563 raft_consensus.cc:729] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2508dd109430caa488ea8de6faf64, State: Initialized, Role: FOLLOWER
I20260805 22:47:28.211573 11563 consensus_queue.cc:260] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } }
I20260805 22:47:28.211536 11560 consensus_queue.cc:260] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } }
I20260805 22:47:28.212050 11562 raft_consensus.cc:348] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } }
I20260805 22:47:28.212554 11562 raft_consensus.cc:374] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:28.212656 11562 raft_consensus.cc:729] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc84e50f73a40dfa3f145e7edf8764f, State: Initialized, Role: FOLLOWER
I20260805 22:47:28.213596 11563 ts_tablet_manager.cc:1428] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260805 22:47:28.213421 11562 consensus_queue.cc:260] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } }
I20260805 22:47:28.213930 11560 ts_tablet_manager.cc:1428] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260805 22:47:28.215551 11562 ts_tablet_manager.cc:1428] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260805 22:47:28.215914 11526 heartbeater.cc:503] Master 127.4.28.62:45821 was elected leader, sending a full tablet report...
I20260805 22:47:28.216048 11343 heartbeater.cc:503] Master 127.4.28.62:45821 was elected leader, sending a full tablet report...
W20260805 22:47:28.355697 11528 tablet.cc:2369] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:47:28.359272 11570 raft_consensus.cc:482] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:28.359579 11570 raft_consensus.cc:504] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } }
I20260805 22:47:28.361555 11570 leader_election.cc:288] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 165eccd0e67340b9b3c79b74ff10611e (127.4.28.2:42529), cbc84e50f73a40dfa3f145e7edf8764f (127.4.28.1:42129)
I20260805 22:47:28.364367 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.362015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42129 (local address 127.4.28.3:34653)
0805 22:47:28.362288 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.362306 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:28.362428 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:28.362769 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:28.362774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:28.362799 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:28.363501 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:47:28.363521 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:28.363542 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:47:28.363594 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:28.363963 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:28.363968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:28.363972 (+ 4us) client_negotiation.cc:769] Sending connection context
0805 22:47:28.364071 (+ 99us) client_negotiation.cc:240] Negotiation successful
0805 22:47:28.364128 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260805 22:47:28.364391 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.362016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:34653 (local address 127.4.28.1:42129)
0805 22:47:28.362192 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.362198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.362387 (+ 189us) server_negotiation.cc:407] Connection header received
0805 22:47:28.362557 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.362563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.362617 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.362705 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.363766 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.363772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.363777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.363798 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.363842 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.363859 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.364033 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.364075 (+ 42us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.364127 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 22:47:28.365371 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.362397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42529 (local address 127.4.28.3:35325)
0805 22:47:28.363110 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.363132 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:28.363252 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:28.363597 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:28.363601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:28.363620 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:28.364323 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:47:28.364326 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:28.364343 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:47:28.364395 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:28.364880 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:28.364885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:28.364888 (+ 3us) client_negotiation.cc:769] Sending connection context
0805 22:47:28.364997 (+ 109us) client_negotiation.cc:240] Negotiation successful
0805 22:47:28.365040 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":391,"threads_started":1}
I20260805 22:47:28.365362 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.361922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:35325 (local address 127.4.28.2:42529)
0805 22:47:28.362127 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.362134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.363228 (+ 1094us) server_negotiation.cc:407] Connection header received
0805 22:47:28.363376 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.363381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.363448 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.363547 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.364601 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.364608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.364612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.364634 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.364675 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.364693 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.364787 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.365020 (+ 233us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.365091 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 22:47:28.365569 11079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6a2308ee384bc4b0c1daef92324314" candidate_uuid: "fef2508dd109430caa488ea8de6faf64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" is_pre_election: true
I20260805 22:47:28.366104 11079 raft_consensus.cc:2454] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2508dd109430caa488ea8de6faf64 in term 0.
I20260805 22:47:28.366822 11369 leader_election.cc:302] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbc84e50f73a40dfa3f145e7edf8764f, fef2508dd109430caa488ea8de6faf64; no voters:
I20260805 22:47:28.367237 11570 raft_consensus.cc:2781] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:28.367357 11570 raft_consensus.cc:482] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:28.367218 11283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6a2308ee384bc4b0c1daef92324314" candidate_uuid: "fef2508dd109430caa488ea8de6faf64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165eccd0e67340b9b3c79b74ff10611e" is_pre_election: true
I20260805 22:47:28.367449 11570 raft_consensus.cc:3037] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:28.367750 11283 raft_consensus.cc:2454] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2508dd109430caa488ea8de6faf64 in term 0.
I20260805 22:47:28.369974 11570 raft_consensus.cc:504] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } }
I20260805 22:47:28.370641 11570 leader_election.cc:288] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [CANDIDATE]: Term 1 election: Requested vote from peers 165eccd0e67340b9b3c79b74ff10611e (127.4.28.2:42529), cbc84e50f73a40dfa3f145e7edf8764f (127.4.28.1:42129)
I20260805 22:47:28.371160 11283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6a2308ee384bc4b0c1daef92324314" candidate_uuid: "fef2508dd109430caa488ea8de6faf64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165eccd0e67340b9b3c79b74ff10611e"
I20260805 22:47:28.371193 11079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6a2308ee384bc4b0c1daef92324314" candidate_uuid: "fef2508dd109430caa488ea8de6faf64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc84e50f73a40dfa3f145e7edf8764f"
I20260805 22:47:28.371369 11283 raft_consensus.cc:3037] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:28.371374 11079 raft_consensus.cc:3037] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:28.373741 11079 raft_consensus.cc:2454] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2508dd109430caa488ea8de6faf64 in term 1.
I20260805 22:47:28.373741 11283 raft_consensus.cc:2454] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2508dd109430caa488ea8de6faf64 in term 1.
I20260805 22:47:28.374202 11369 leader_election.cc:302] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbc84e50f73a40dfa3f145e7edf8764f, fef2508dd109430caa488ea8de6faf64; no voters:
I20260805 22:47:28.374513 11570 raft_consensus.cc:2781] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:28.374991 11570 raft_consensus.cc:686] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 1 LEADER]: Becoming Leader. State: Replica: fef2508dd109430caa488ea8de6faf64, State: Running, Role: LEADER
I20260805 22:47:28.375475 11570 consensus_queue.cc:237] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } }
W20260805 22:47:28.380182 11344 tablet.cc:2369] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:47:28.380089 10874 catalog_manager.cc:5495] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef2508dd109430caa488ea8de6faf64 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "fef2508dd109430caa488ea8de6faf64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2508dd109430caa488ea8de6faf64" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 34621 } health_report { overall_health: HEALTHY } } }
I20260805 22:47:28.413057 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.410897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.4.28.3:34621)
0805 22:47:28.411110 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.411115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.411198 (+ 83us) server_negotiation.cc:407] Connection header received
0805 22:47:28.411385 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.411389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.411477 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.411586 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.412428 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.412432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.412436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.412449 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.412478 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.412489 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.412601 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.412753 (+ 152us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.412804 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260805 22:47:28.420666 11152 tablet.cc:2369] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:47:28.422217 11079 raft_consensus.cc:1261] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [term 1 FOLLOWER]: Refusing update from remote peer fef2508dd109430caa488ea8de6faf64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:28.422219 11283 raft_consensus.cc:1261] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [term 1 FOLLOWER]: Refusing update from remote peer fef2508dd109430caa488ea8de6faf64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:28.423382 11570 consensus_queue.cc:1035] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc84e50f73a40dfa3f145e7edf8764f" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:28.423703 11574 consensus_queue.cc:1035] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "165eccd0e67340b9b3c79b74ff10611e" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:28.437672 11580 mvcc.cc:205] Tried to move back new op lower bound from 7315333318326321152 to 7315333318149922816. Current Snapshot: MvccSnapshot[applied={T|T < 7315333318326321152}]
I20260805 22:47:28.438444 11579 mvcc.cc:205] Tried to move back new op lower bound from 7315333318326321152 to 7315333318149922816. Current Snapshot: MvccSnapshot[applied={T|T < 7315333318326321152}]
I20260805 22:47:28.454883 10874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:28.455243 10874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:28.459560 10874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c4425d814a343f49a3b98a19ff3290c: Sending DeleteTablet for 3 replicas of tablet fa6a2308ee384bc4b0c1daef92324314
I20260805 22:47:28.460951 11249 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6a2308ee384bc4b0c1daef92324314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:28 UTC) from {username='slave'} at 127.0.0.1:36078
I20260805 22:47:28.461122 11049 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6a2308ee384bc4b0c1daef92324314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:28 UTC) from {username='slave'} at 127.0.0.1:57892
I20260805 22:47:28.461839 11438 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6a2308ee384bc4b0c1daef92324314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:28 UTC) from {username='slave'} at 127.0.0.1:38592
I20260805 22:47:28.462994 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 10960
I20260805 22:47:28.465245 11596 tablet_replica.cc:330] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: stopping tablet replica
I20260805 22:47:28.465902 11596 raft_consensus.cc:2229] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:28.466756 11596 raft_consensus.cc:2258] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:28.471472 11592 tablet_replica.cc:330] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: stopping tablet replica
I20260805 22:47:28.472154 11595 tablet_replica.cc:330] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f: stopping tablet replica
I20260805 22:47:28.472385 11596 ts_tablet_manager.cc:1905] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:28.472409 11592 raft_consensus.cc:2229] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:28.472738 11595 raft_consensus.cc:2229] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:28.473140 11592 raft_consensus.cc:2258] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:28.473346 11595 raft_consensus.cc:2258] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:28.475900 11595 ts_tablet_manager.cc:1905] T fa6a2308ee384bc4b0c1daef92324314 P cbc84e50f73a40dfa3f145e7edf8764f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:28.475901 11592 ts_tablet_manager.cc:1905] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:28.477130 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.464133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.4.28.1:42129)
0805 22:47:28.464358 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.464365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.474386 (+ 10021us) server_negotiation.cc:407] Connection header received
0805 22:47:28.474668 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.474674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.474774 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.474858 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.476251 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.476257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.476261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.476285 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.476325 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.476342 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.476471 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.476746 (+ 275us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.476816 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260805 22:47:28.478448 11144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:28.485594 11596 ts_tablet_manager.cc:1918] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:28.485951 11596 log.cc:1196] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/fa6a2308ee384bc4b0c1daef92324314
I20260805 22:47:28.486722 11596 ts_tablet_manager.cc:1939] T fa6a2308ee384bc4b0c1daef92324314 P fef2508dd109430caa488ea8de6faf64: Deleting consensus metadata
I20260805 22:47:28.488592 11592 ts_tablet_manager.cc:1918] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:28.488889 11592 log.cc:1196] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/fa6a2308ee384bc4b0c1daef92324314
I20260805 22:47:28.489490 11592 ts_tablet_manager.cc:1939] T fa6a2308ee384bc4b0c1daef92324314 P 165eccd0e67340b9b3c79b74ff10611e: Deleting consensus metadata
I20260805 22:47:28.489508 10851 catalog_manager.cc:4841] TS fef2508dd109430caa488ea8de6faf64 (127.4.28.3:34621): tablet fa6a2308ee384bc4b0c1daef92324314 (table test-table [id=c88607fd699444f48dba12990f9bc730]) successfully deleted
I20260805 22:47:28.491760 10855 catalog_manager.cc:4841] TS 165eccd0e67340b9b3c79b74ff10611e (127.4.28.2:42529): tablet fa6a2308ee384bc4b0c1daef92324314 (table test-table [id=c88607fd699444f48dba12990f9bc730]) successfully deleted
I20260805 22:47:28.576637 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 10960
W20260805 22:47:28.609972 10851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.28.1:42129 (error 108)
I20260805 22:47:28.610282 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 11157
I20260805 22:47:28.613155 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.612611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42129 (local address 127.0.0.1:57918)
0805 22:47:28.612794 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.613006 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260805 22:47:28.613425 10851 catalog_manager.cc:4584] TS cbc84e50f73a40dfa3f145e7edf8764f (127.4.28.1:42129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa6a2308ee384bc4b0c1daef92324314: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42129: connect: Connection refused (error 111)
I20260805 22:47:28.614481 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.611293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.4.28.2:42529)
0805 22:47:28.611509 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.611516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.611543 (+ 27us) server_negotiation.cc:407] Connection header received
0805 22:47:28.611759 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.611766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.611898 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.612007 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.613743 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.613751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.613756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.613779 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.613823 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.613839 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.613977 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.614105 (+ 128us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.614174 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260805 22:47:28.615253 11334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:28.667757 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.667174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42129 (local address 127.0.0.1:57924)
0805 22:47:28.667434 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.667566 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260805 22:47:28.669055 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.668614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42129 (local address 127.0.0.1:57938)
0805 22:47:28.668790 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.668886 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260805 22:47:28.703477 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 11157
I20260805 22:47:28.727802 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 11346
I20260805 22:47:28.729786 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.729254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42129 (local address 127.0.0.1:57948)
0805 22:47:28.729438 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.729562 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260805 22:47:28.731022 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.728624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.4.28.3:34621)
0805 22:47:28.728808 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:28.728819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:28.728879 (+ 60us) server_negotiation.cc:407] Connection header received
0805 22:47:28.729118 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:28.729122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:28.729183 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:28.729279 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:28.730269 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:28.730275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:28.730280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:28.730306 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:28.730345 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:28.730362 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:28.730520 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:28.730672 (+ 152us) server_negotiation.cc:299] Negotiation successful
0805 22:47:28.730723 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260805 22:47:28.731505 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.731108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42129 (local address 127.0.0.1:57964)
0805 22:47:28.731246 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.731349 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260805 22:47:28.731875 11520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:28.812247 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.811714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42129 (local address 127.0.0.1:57966)
0805 22:47:28.811963 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.812063 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260805 22:47:28.813341 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:28.812932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42129 (local address 127.0.0.1:57982)
0805 22:47:28.813138 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:28.813211 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260805 22:47:28.854194 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 11346
I20260805 22:47:28.879043 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 10828
I20260805 22:47:28.880414 10919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:28.987818 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 10828
2026-08-05T22:47:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2925 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260805 22:47:29.035807 4208 mini_oidc.cc:152] Starting JWKS server
I20260805 22:47:29.038892 4208 webserver.cc:460] Webserver started at https://127.0.0.1:33817/ using document root <none> and password file <none>
I20260805 22:47:29.039299 4208 mini_oidc.cc:189] Starting OIDC Discovery server
I20260805 22:47:29.040005 4208 webserver.cc:460] Webserver started at http://127.0.0.1:45125/ using document root <none> and password file <none>
2026-08-05T22:47:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:29Z Disabled control of system clock
I20260805 22:47:29.078161 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37863
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:37863
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33817/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/testchainca.cert with env {}
W20260805 22:47:29.428053 11617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:29.428391 11617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:29.428526 11617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:29.436138 11617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:29.436291 11617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:29.436359 11617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:29.436450 11617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:29.436519 11617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260805 22:47:29.436606 11617 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260805 22:47:29.436657 11617 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33817/jwks.json
I20260805 22:47:29.447206 11617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:37863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33817/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:29.448662 11617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:29.450153 11617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:29.463229 11624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:29.463197 11626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:29.463263 11628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:29.464393 11617 server_base.cc:1034] running on GCE node
I20260805 22:47:29.465356 11617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:29.467069 11617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:29.468343 11617 hybrid_clock.cc:648] HybridClock initialized: now 1785970049468285 us; error 43 us; skew 500 ppm
I20260805 22:47:29.468740 11617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:47:29.471320 11617 webserver.cc:460] Webserver started at http://127.4.28.62:36769/ using document root <none> and password file <none>
I20260805 22:47:29.472107 11617 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:29.472230 11617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:29.472581 11617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:29.475862 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "a74cf69d99d44d318a594133a072de59"
format_stamp: "Formatted at 2026-08-05 22:47:29 on dist-test-slave-7lbf"
server_key: "824ab62ad82f5575aabdbe47a2a48b6d"
server_key_iv: "8466d9c8509f27f6fb91a6f3d7c516a0"
server_key_version: "encryptionkey@0"
I20260805 22:47:29.476756 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "a74cf69d99d44d318a594133a072de59"
format_stamp: "Formatted at 2026-08-05 22:47:29 on dist-test-slave-7lbf"
server_key: "824ab62ad82f5575aabdbe47a2a48b6d"
server_key_iv: "8466d9c8509f27f6fb91a6f3d7c516a0"
server_key_version: "encryptionkey@0"
I20260805 22:47:29.482575 11617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260805 22:47:29.486980 11635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:29.488597 11617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260805 22:47:29.488816 11617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "a74cf69d99d44d318a594133a072de59"
format_stamp: "Formatted at 2026-08-05 22:47:29 on dist-test-slave-7lbf"
server_key: "824ab62ad82f5575aabdbe47a2a48b6d"
server_key_iv: "8466d9c8509f27f6fb91a6f3d7c516a0"
server_key_version: "encryptionkey@0"
I20260805 22:47:29.489073 11617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:29.527938 11617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:29.528934 11617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:29.529238 11617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:29.548731 11617 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:37863
I20260805 22:47:29.548734 11707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:37863 every 8 connection(s)
I20260805 22:47:29.550232 11617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:29.554360 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 11617
I20260805 22:47:29.554451 11709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:29.554543 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:29.555197 4208 external_mini_cluster.cc:1428] Setting key a8609c00f2057f5f8097946d888ea147
I20260805 22:47:29.569753 11709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59: Bootstrap starting.
I20260805 22:47:29.572743 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:29.557298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.4.28.62:37863)
0805 22:47:29.558569 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:29.558586 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:29.558646 (+ 60us) server_negotiation.cc:407] Connection header received
0805 22:47:29.561534 (+ 2888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:29.561574 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:29.562340 (+ 766us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:29.562885 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:29.563479 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:29.565401 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:29.567001 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:29.568101 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:29.568302 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:29.569629 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:29.569641 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:29.569662 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:29.570414 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:29.570576 (+ 162us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:29.570607 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:29.570779 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:29.570984 (+ 205us) server_negotiation.cc:299] Negotiation successful
0805 22:47:29.571346 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":346,"threads_started":1}
I20260805 22:47:29.573980 11709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:29.575129 11709 log.cc:824] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:29.577219 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:47:29.578501 11709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59: No bootstrap required, opened a new log
I20260805 22:47:29.586572 11709 raft_consensus.cc:348] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74cf69d99d44d318a594133a072de59" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37863 } }
I20260805 22:47:29.587035 11709 raft_consensus.cc:374] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:29.587123 11709 raft_consensus.cc:729] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74cf69d99d44d318a594133a072de59, State: Initialized, Role: FOLLOWER
I20260805 22:47:29.587710 11709 consensus_queue.cc:260] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74cf69d99d44d318a594133a072de59" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37863 } }
I20260805 22:47:29.587925 11709 raft_consensus.cc:388] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:29.588027 11709 raft_consensus.cc:482] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:29.588160 11709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:29.590390 11709 raft_consensus.cc:504] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74cf69d99d44d318a594133a072de59" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37863 } }
I20260805 22:47:29.590921 11709 leader_election.cc:302] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a74cf69d99d44d318a594133a072de59; no voters:
I20260805 22:47:29.591488 11709 leader_election.cc:288] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:29.591634 11718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:29.592334 11718 raft_consensus.cc:686] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [term 1 LEADER]: Becoming Leader. State: Replica: a74cf69d99d44d318a594133a072de59, State: Running, Role: LEADER
I20260805 22:47:29.592914 11718 consensus_queue.cc:237] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74cf69d99d44d318a594133a072de59" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37863 } }
I20260805 22:47:29.593744 11709 sys_catalog.cc:564] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:29.596176 11719 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a74cf69d99d44d318a594133a072de59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74cf69d99d44d318a594133a072de59" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37863 } } }
I20260805 22:47:29.596683 11719 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:29.597355 11723 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a74cf69d99d44d318a594133a072de59. Latest consensus state: current_term: 1 leader_uuid: "a74cf69d99d44d318a594133a072de59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74cf69d99d44d318a594133a072de59" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37863 } } }
I20260805 22:47:29.597692 11723 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:29.598766 11734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:29.605916 11734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:29.616137 11734 catalog_manager.cc:1347] Generated new cluster ID: bc23b049921e44d7a701227ab0149c06
I20260805 22:47:29.616290 11734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:29.642060 11734 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:29.643442 11734 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:29.649827 11734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a74cf69d99d44d318a594133a072de59: Generated new TSK 0
I20260805 22:47:29.650898 11734 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260805 22:47:29.711083 11757 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260805 22:47:29.711351 11757 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260805 22:47:29.712152 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:29.682054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.4.28.62:37863)
0805 22:47:29.682310 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:29.682316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:29.682812 (+ 496us) server_negotiation.cc:407] Connection header received
0805 22:47:29.683182 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:29.683187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:29.683279 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:29.683412 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0805 22:47:29.684109 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:29.684904 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:29.687144 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:29.687646 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:29.687761 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:29.688488 (+ 727us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0805 22:47:29.711544 (+ 23056us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0805 22:47:29.711911 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":83}
W20260805 22:47:29.712281 11713 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260805 22:47:29.714550 11756 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:29.681891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37863 (local address 127.0.0.1:40750)
0805 22:47:29.682662 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:29.682685 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:29.682922 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:29.683481 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:29.683487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:29.683502 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0805 22:47:29.683916 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:29.683929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:29.685058 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:29.685064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:29.686970 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:29.686990 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:29.687783 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:29.687860 (+ 77us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:29.688072 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:29.688167 (+ 95us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0805 22:47:29.711903 (+ 23736us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0805 22:47:29.712314 (+ 411us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.28.62:37863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":389,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.28.62:37863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260805 22:47:29.716253 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 11617
I20260805 22:47:29.717674 11702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:29.813903 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 11617
2026-08-05T22:47:29Z chronyd exiting
I20260805 22:47:29.865410 4208 test_util.cc:175] -----------------------------------------------
I20260805 22:47:29.865621 4208 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785970002413767-4208-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (834 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260805 22:47:29.870868 4208 mini_oidc.cc:152] Starting JWKS server
I20260805 22:47:29.872850 4208 webserver.cc:460] Webserver started at https://127.0.0.1:34307/ using document root <none> and password file <none>
I20260805 22:47:29.873127 4208 mini_oidc.cc:189] Starting OIDC Discovery server
I20260805 22:47:29.873883 4208 webserver.cc:460] Webserver started at http://127.0.0.1:43741/ using document root <none> and password file <none>
2026-08-05T22:47:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:29Z Disabled control of system clock
I20260805 22:47:29.911326 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:38993
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:38993
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34307/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/testchainca.cert with env {}
W20260805 22:47:30.264542 11765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:30.264906 11765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:30.265056 11765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:30.272756 11765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:30.272933 11765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:30.273005 11765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:30.273085 11765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:30.273149 11765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260805 22:47:30.273236 11765 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260805 22:47:30.273291 11765 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34307/jwks.json
I20260805 22:47:30.284029 11765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:38993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:38993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34307/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:30.285554 11765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:30.287021 11765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:30.299463 11771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:30.299373 11772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:30.299419 11775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:30.300534 11765 server_base.cc:1034] running on GCE node
I20260805 22:47:30.301388 11765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:30.303058 11765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:30.304289 11765 hybrid_clock.cc:648] HybridClock initialized: now 1785970050304232 us; error 54 us; skew 500 ppm
I20260805 22:47:30.304677 11765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:47:30.307190 11765 webserver.cc:460] Webserver started at http://127.4.28.62:34513/ using document root <none> and password file <none>
I20260805 22:47:30.307926 11765 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:30.308041 11765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:30.308374 11765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:30.311553 11765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "c7dead0d4eb3437f9b967675c7a1c461"
format_stamp: "Formatted at 2026-08-05 22:47:30 on dist-test-slave-7lbf"
server_key: "bf94e2882f45d8ccccd0c73268c6db2e"
server_key_iv: "3e769f15358c1f0b559fb67d47a64ed1"
server_key_version: "encryptionkey@0"
I20260805 22:47:30.312433 11765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "c7dead0d4eb3437f9b967675c7a1c461"
format_stamp: "Formatted at 2026-08-05 22:47:30 on dist-test-slave-7lbf"
server_key: "bf94e2882f45d8ccccd0c73268c6db2e"
server_key_iv: "3e769f15358c1f0b559fb67d47a64ed1"
server_key_version: "encryptionkey@0"
I20260805 22:47:30.318054 11765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260805 22:47:30.321800 11784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:30.323235 11765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:30.323412 11765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "c7dead0d4eb3437f9b967675c7a1c461"
format_stamp: "Formatted at 2026-08-05 22:47:30 on dist-test-slave-7lbf"
server_key: "bf94e2882f45d8ccccd0c73268c6db2e"
server_key_iv: "3e769f15358c1f0b559fb67d47a64ed1"
server_key_version: "encryptionkey@0"
I20260805 22:47:30.323635 11765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:30.346606 11765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:30.347541 11765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:30.347843 11765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:30.366449 11765 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:38993
I20260805 22:47:30.366451 11869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:38993 every 8 connection(s)
I20260805 22:47:30.367987 11765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:30.372148 11870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:30.377885 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 11765
I20260805 22:47:30.378122 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:30.378749 4208 external_mini_cluster.cc:1428] Setting key 95bec8a2056ff2e6e6faed1842ecf104
I20260805 22:47:30.384063 11870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461: Bootstrap starting.
I20260805 22:47:30.387609 11870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:30.388785 11870 log.cc:824] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:30.392091 11870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461: No bootstrap required, opened a new log
I20260805 22:47:30.393457 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:30.380889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.4.28.62:38993)
0805 22:47:30.382109 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:30.382122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:30.382173 (+ 51us) server_negotiation.cc:407] Connection header received
0805 22:47:30.384004 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:30.384033 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:30.384641 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:30.385087 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:30.385762 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:30.387126 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:30.388576 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:30.389391 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:30.389655 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:30.390860 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:30.390869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:30.390888 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:30.391489 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:30.391579 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:30.391603 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:30.391713 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:30.391994 (+ 281us) server_negotiation.cc:299] Negotiation successful
0805 22:47:30.392385 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":388,"threads_started":1}
I20260805 22:47:30.397989 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:47:30.400430 11870 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dead0d4eb3437f9b967675c7a1c461" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38993 } }
I20260805 22:47:30.400902 11870 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:30.401021 11870 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dead0d4eb3437f9b967675c7a1c461, State: Initialized, Role: FOLLOWER
I20260805 22:47:30.401798 11870 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dead0d4eb3437f9b967675c7a1c461" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38993 } }
I20260805 22:47:30.402061 11870 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:30.402195 11870 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:30.402354 11870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:30.404879 11870 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dead0d4eb3437f9b967675c7a1c461" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38993 } }
I20260805 22:47:30.405485 11870 leader_election.cc:302] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7dead0d4eb3437f9b967675c7a1c461; no voters:
I20260805 22:47:30.406173 11870 leader_election.cc:288] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:30.406359 11876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:30.407793 11876 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [term 1 LEADER]: Becoming Leader. State: Replica: c7dead0d4eb3437f9b967675c7a1c461, State: Running, Role: LEADER
I20260805 22:47:30.408380 11876 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dead0d4eb3437f9b967675c7a1c461" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38993 } }
I20260805 22:47:30.408609 11870 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:30.411381 11879 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7dead0d4eb3437f9b967675c7a1c461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dead0d4eb3437f9b967675c7a1c461" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38993 } } }
I20260805 22:47:30.411888 11879 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:30.412413 11880 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7dead0d4eb3437f9b967675c7a1c461. Latest consensus state: current_term: 1 leader_uuid: "c7dead0d4eb3437f9b967675c7a1c461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dead0d4eb3437f9b967675c7a1c461" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38993 } } }
I20260805 22:47:30.412738 11880 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:30.414043 11885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:30.419289 11885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:30.428726 11885 catalog_manager.cc:1347] Generated new cluster ID: 30184638d22d4e5792ab17ab1d4feb37
I20260805 22:47:30.428864 11885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:30.440385 11885 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:30.441846 11885 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:30.452562 11885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7dead0d4eb3437f9b967675c7a1c461: Generated new TSK 0
I20260805 22:47:30.453480 11885 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260805 22:47:30.492149 11908 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260805 22:47:30.492297 11908 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260805 22:47:30.493759 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:30.472506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.4.28.62:38993)
0805 22:47:30.472729 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:30.472736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:30.473081 (+ 345us) server_negotiation.cc:407] Connection header received
0805 22:47:30.473327 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:30.473337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:30.473441 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:30.473701 (+ 260us) server_negotiation.cc:226] Negotiated authn=JWT
0805 22:47:30.474185 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:30.475357 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:30.476874 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:30.477640 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:30.477782 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:30.478414 (+ 632us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0805 22:47:30.492871 (+ 14457us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0805 22:47:30.493410 (+ 539us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33566: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":89}
W20260805 22:47:30.493965 11907 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:30.472312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:38993 (local address 127.0.0.1:33566)
0805 22:47:30.472962 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:30.472979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:30.473125 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:30.473701 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:30.473705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:30.473715 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0805 22:47:30.474018 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:30.474029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:30.475568 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:30.475573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:30.476709 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:30.476723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:30.477813 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:30.477870 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:30.478040 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:30.478081 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0805 22:47:30.493216 (+ 15135us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0805 22:47:30.493565 (+ 349us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.28.62:38993: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":348,"threads_started":1}
W20260805 22:47:30.494128 11874 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33566: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260805 22:47:30.495410 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 11765
I20260805 22:47:30.496783 11860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:30.591272 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 11765
2026-08-05T22:47:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (784 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260805 22:47:30.653900 4208 mini_oidc.cc:152] Starting JWKS server
I20260805 22:47:30.655918 4208 webserver.cc:460] Webserver started at https://127.0.0.1:46159/ using document root <none> and password file <none>
I20260805 22:47:30.656173 4208 mini_oidc.cc:189] Starting OIDC Discovery server
I20260805 22:47:30.656885 4208 webserver.cc:460] Webserver started at http://127.0.0.1:33687/ using document root <none> and password file <none>
2026-08-05T22:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:30Z Disabled control of system clock
I20260805 22:47:30.695170 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45143
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45143
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46159/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260805 22:47:31.053792 11919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:31.054157 11919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:31.054294 11919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:31.062590 11919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:31.062750 11919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:31.062848 11919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:31.062938 11919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:31.063000 11919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260805 22:47:31.063086 11919 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260805 22:47:31.063140 11919 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46159/jwks.json
I20260805 22:47:31.073203 11919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46159/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:31.074641 11919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:31.076574 11919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:31.089100 11924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:31.089118 11928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:31.089177 11926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:31.090219 11919 server_base.cc:1034] running on GCE node
I20260805 22:47:31.091114 11919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:31.092756 11919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:31.094026 11919 hybrid_clock.cc:648] HybridClock initialized: now 1785970051093960 us; error 64 us; skew 500 ppm
I20260805 22:47:31.094426 11919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:47:31.096935 11919 webserver.cc:460] Webserver started at http://127.4.28.62:33077/ using document root <none> and password file <none>
I20260805 22:47:31.097743 11919 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:31.097855 11919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:31.098201 11919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:31.101333 11919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "5f7686f43a2d40c8a4768975d17cac89"
format_stamp: "Formatted at 2026-08-05 22:47:31 on dist-test-slave-7lbf"
server_key: "99259657649bdef1c1db3bf7dcd30c2c"
server_key_iv: "21a605079c98b6d3963445c854cbd2dd"
server_key_version: "encryptionkey@0"
I20260805 22:47:31.102347 11919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "5f7686f43a2d40c8a4768975d17cac89"
format_stamp: "Formatted at 2026-08-05 22:47:31 on dist-test-slave-7lbf"
server_key: "99259657649bdef1c1db3bf7dcd30c2c"
server_key_iv: "21a605079c98b6d3963445c854cbd2dd"
server_key_version: "encryptionkey@0"
I20260805 22:47:31.108875 11919 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260805 22:47:31.113443 11939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:31.115438 11919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:47:31.115669 11919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "5f7686f43a2d40c8a4768975d17cac89"
format_stamp: "Formatted at 2026-08-05 22:47:31 on dist-test-slave-7lbf"
server_key: "99259657649bdef1c1db3bf7dcd30c2c"
server_key_iv: "21a605079c98b6d3963445c854cbd2dd"
server_key_version: "encryptionkey@0"
I20260805 22:47:31.115914 11919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:31.152354 11919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:31.153319 11919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:31.153666 11919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:31.172330 11919 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:45143
I20260805 22:47:31.172348 12021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:45143 every 8 connection(s)
I20260805 22:47:31.173834 11919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:31.177888 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:31.182510 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 11919
I20260805 22:47:31.182721 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:31.183401 4208 external_mini_cluster.cc:1428] Setting key b30fbc7d4eb1f4dbebf111ddf6f92606
I20260805 22:47:31.189476 12022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89: Bootstrap starting.
I20260805 22:47:31.192901 12022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:31.194132 12022 log.cc:824] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:31.197901 12022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89: No bootstrap required, opened a new log
I20260805 22:47:31.198092 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:31.185422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.4.28.62:45143)
0805 22:47:31.186614 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:31.186628 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:31.186682 (+ 54us) server_negotiation.cc:407] Connection header received
0805 22:47:31.188462 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:31.188490 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:31.189027 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:31.189426 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:31.190064 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:31.191428 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:31.192815 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:31.193655 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:31.193846 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:31.195154 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:31.195162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:31.195178 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:31.195839 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:31.195948 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:31.195978 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:31.196107 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:31.196354 (+ 247us) server_negotiation.cc:299] Negotiation successful
0805 22:47:31.196716 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":405,"threads_started":1}
I20260805 22:47:31.202270 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:47:31.207019 12022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7686f43a2d40c8a4768975d17cac89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45143 } }
I20260805 22:47:31.207412 12022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:31.207490 12022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7686f43a2d40c8a4768975d17cac89, State: Initialized, Role: FOLLOWER
I20260805 22:47:31.208042 12022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7686f43a2d40c8a4768975d17cac89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45143 } }
I20260805 22:47:31.208266 12022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:31.208412 12022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:31.208595 12022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:31.211021 12022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7686f43a2d40c8a4768975d17cac89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45143 } }
I20260805 22:47:31.211620 12022 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f7686f43a2d40c8a4768975d17cac89; no voters:
I20260805 22:47:31.212337 12022 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:31.212500 12029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:31.213330 12029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [term 1 LEADER]: Becoming Leader. State: Replica: 5f7686f43a2d40c8a4768975d17cac89, State: Running, Role: LEADER
I20260805 22:47:31.213920 12029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7686f43a2d40c8a4768975d17cac89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45143 } }
I20260805 22:47:31.214700 12022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:31.216988 12030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f7686f43a2d40c8a4768975d17cac89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7686f43a2d40c8a4768975d17cac89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45143 } } }
I20260805 22:47:31.217336 12030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:31.217720 12031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7686f43a2d40c8a4768975d17cac89. Latest consensus state: current_term: 1 leader_uuid: "5f7686f43a2d40c8a4768975d17cac89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7686f43a2d40c8a4768975d17cac89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45143 } } }
I20260805 22:47:31.219362 12031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:31.220500 12035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:31.226308 12035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:31.236259 12035 catalog_manager.cc:1347] Generated new cluster ID: 5a69f183211e487f9fc070db22939beb
I20260805 22:47:31.236407 12035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:31.254036 12035 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:31.255447 12035 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:31.270016 12035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f7686f43a2d40c8a4768975d17cac89: Generated new TSK 0
I20260805 22:47:31.270983 12035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:47:31.295163 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:31.285964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.4.28.62:45143)
0805 22:47:31.286227 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:31.286234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:31.286674 (+ 440us) server_negotiation.cc:407] Connection header received
0805 22:47:31.287003 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:31.287008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:31.287089 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:31.287249 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0805 22:47:31.287835 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:31.288616 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:31.290750 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:31.291214 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:31.291336 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:31.291990 (+ 654us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0805 22:47:31.294485 (+ 2495us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46159/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0805 22:47:31.294838 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46159/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":78}
W20260805 22:47:31.295296 12025 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46159/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260805 22:47:31.297224 12059 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:31.285785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45143 (local address 127.0.0.1:49698)
0805 22:47:31.286542 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:31.286563 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:31.286746 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:31.287287 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:31.287292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:31.287306 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0805 22:47:31.287641 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:31.287653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:31.288776 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:31.288782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:31.290533 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:31.290568 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:31.291354 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:31.291439 (+ 85us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:31.291652 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:31.291705 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0805 22:47:31.294734 (+ 3029us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46159/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0805 22:47:31.296751 (+ 2017us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.28.62:45143: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46159/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":1622,"thread_start_us":359,"threads_started":1}
I20260805 22:47:31.298599 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 11919
I20260805 22:47:31.299955 12016 generic_service.cc:196] Checking for leaks (request via RPC)
E20260805 22:47:31.307442 12060 webserver.cc:533] Webserver: sslize failed:
E20260805 22:47:31.307554 12060 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260805 22:47:31.393838 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 11919
2026-08-05T22:47:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (802 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-05T22:47:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:31Z Disabled control of system clock
I20260805 22:47:31.493064 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:35221
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:35221
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/testchainca.cert with env {}
W20260805 22:47:31.847640 12068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:31.848006 12068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:31.848152 12068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:31.856190 12068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:31.856355 12068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:31.856431 12068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:31.856511 12068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:31.856580 12068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260805 22:47:31.856674 12068 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260805 22:47:31.856732 12068 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260805 22:47:31.868005 12068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:35221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:31.869589 12068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:31.871109 12068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:31.884464 12077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:31.884593 12074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:31.884557 12084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:31.885177 12068 server_base.cc:1034] running on GCE node
I20260805 22:47:31.886204 12068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:31.888008 12068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:31.889303 12068 hybrid_clock.cc:648] HybridClock initialized: now 1785970051889237 us; error 48 us; skew 500 ppm
I20260805 22:47:31.889787 12068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:47:31.892445 12068 webserver.cc:460] Webserver started at http://127.4.28.62:32769/ using document root <none> and password file <none>
I20260805 22:47:31.893244 12068 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:31.893383 12068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:31.893774 12068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:31.897023 12068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "e18376815a3c41669827d85ce7ce3f80"
format_stamp: "Formatted at 2026-08-05 22:47:31 on dist-test-slave-7lbf"
server_key: "2c95f08aed6b60e50cbb408f4527813d"
server_key_iv: "4ea7f9f5163489dac743ff035c2ab208"
server_key_version: "encryptionkey@0"
I20260805 22:47:31.897984 12068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "e18376815a3c41669827d85ce7ce3f80"
format_stamp: "Formatted at 2026-08-05 22:47:31 on dist-test-slave-7lbf"
server_key: "2c95f08aed6b60e50cbb408f4527813d"
server_key_iv: "4ea7f9f5163489dac743ff035c2ab208"
server_key_version: "encryptionkey@0"
I20260805 22:47:31.903852 12068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260805 22:47:31.908036 12091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:31.909667 12068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 22:47:31.909878 12068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "e18376815a3c41669827d85ce7ce3f80"
format_stamp: "Formatted at 2026-08-05 22:47:31 on dist-test-slave-7lbf"
server_key: "2c95f08aed6b60e50cbb408f4527813d"
server_key_iv: "4ea7f9f5163489dac743ff035c2ab208"
server_key_version: "encryptionkey@0"
I20260805 22:47:31.910128 12068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:31.931958 12068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:31.932925 12068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:31.933239 12068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:31.952329 12068 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:35221
I20260805 22:47:31.952347 12154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:35221 every 8 connection(s)
I20260805 22:47:31.953805 12068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:31.957943 12155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:31.959771 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 12068
I20260805 22:47:31.960044 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:31.960721 4208 external_mini_cluster.cc:1428] Setting key 06bfdaa0c7414acf26916aa56f0dab17
I20260805 22:47:31.970376 12155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80: Bootstrap starting.
I20260805 22:47:31.974435 12155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:31.975694 12155 log.cc:824] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:31.975996 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:31.962982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.4.28.62:35221)
0805 22:47:31.964155 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:31.964168 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:31.964217 (+ 49us) server_negotiation.cc:407] Connection header received
0805 22:47:31.966003 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:31.966036 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:31.966580 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:31.966974 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:31.967706 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:31.969108 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:31.971065 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:31.971796 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:31.972004 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:31.973305 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:31.973314 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:31.973332 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:47:31.974000 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:31.974106 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:47:31.974130 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:31.974225 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:31.974496 (+ 271us) server_negotiation.cc:299] Negotiation successful
0805 22:47:31.974880 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":445,"threads_started":1}
I20260805 22:47:31.979115 12155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80: No bootstrap required, opened a new log
I20260805 22:47:31.980579 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:47:31.987870 12155 raft_consensus.cc:348] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18376815a3c41669827d85ce7ce3f80" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35221 } }
I20260805 22:47:31.988277 12155 raft_consensus.cc:374] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:31.988370 12155 raft_consensus.cc:729] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18376815a3c41669827d85ce7ce3f80, State: Initialized, Role: FOLLOWER
I20260805 22:47:31.989055 12155 consensus_queue.cc:260] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18376815a3c41669827d85ce7ce3f80" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35221 } }
I20260805 22:47:31.989322 12155 raft_consensus.cc:388] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:31.989463 12155 raft_consensus.cc:482] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:31.989670 12155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:31.991981 12155 raft_consensus.cc:504] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18376815a3c41669827d85ce7ce3f80" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35221 } }
I20260805 22:47:31.992666 12155 leader_election.cc:302] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e18376815a3c41669827d85ce7ce3f80; no voters:
I20260805 22:47:31.993467 12155 leader_election.cc:288] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:31.993659 12161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:31.994809 12161 raft_consensus.cc:686] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [term 1 LEADER]: Becoming Leader. State: Replica: e18376815a3c41669827d85ce7ce3f80, State: Running, Role: LEADER
I20260805 22:47:31.995419 12161 consensus_queue.cc:237] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18376815a3c41669827d85ce7ce3f80" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35221 } }
I20260805 22:47:31.996018 12155 sys_catalog.cc:564] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:31.998457 12162 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e18376815a3c41669827d85ce7ce3f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18376815a3c41669827d85ce7ce3f80" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35221 } } }
I20260805 22:47:31.998449 12164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e18376815a3c41669827d85ce7ce3f80. Latest consensus state: current_term: 1 leader_uuid: "e18376815a3c41669827d85ce7ce3f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e18376815a3c41669827d85ce7ce3f80" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35221 } } }
I20260805 22:47:31.998845 12162 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:31.998912 12164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:31.999672 12171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:32.005831 12171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:32.020999 12171 catalog_manager.cc:1347] Generated new cluster ID: fe87bdf5d9e745208f558ed0f27e80a9
I20260805 22:47:32.021155 12171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:32.034343 12171 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:32.035730 12171 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:32.044471 12171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e18376815a3c41669827d85ce7ce3f80: Generated new TSK 0
I20260805 22:47:32.045603 12171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:47:32.525846 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:32.068659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.4.28.62:35221)
0805 22:47:32.068878 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:32.068885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:32.069177 (+ 292us) server_negotiation.cc:407] Connection header received
0805 22:47:32.069418 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:32.069424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:32.069570 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:32.069780 (+ 210us) server_negotiation.cc:226] Negotiated authn=JWT
0805 22:47:32.070367 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:32.071618 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:32.073153 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:32.073897 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:32.074041 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:32.074600 (+ 559us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0805 22:47:32.525209 (+450609us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0805 22:47:32.525609 (+ 400us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52606: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":85}
W20260805 22:47:32.525950 12159 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52606: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260805 22:47:32.526194 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:32.068419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:35221 (local address 127.0.0.1:52606)
0805 22:47:32.069062 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:32.069080 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:32.069223 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:32.069836 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:32.069840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:32.069851 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0805 22:47:32.070189 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:32.070199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:32.071816 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:32.071821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:32.072985 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:32.073000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:32.074048 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:32.074106 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:32.074250 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:32.074289 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0805 22:47:32.525565 (+451276us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0805 22:47:32.525822 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.28.62:35221: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":334,"threads_started":1}
I20260805 22:47:32.527554 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 12068
I20260805 22:47:32.528944 12148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:32.631978 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 12068
2026-08-05T22:47:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1219 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (450 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (447 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:33 dist-test-slave-7lbf krb5kdc[12220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:33 dist-test-slave-7lbf krb5kdc[12220](info): set up 2 sockets
Aug 05 22:47:33 dist-test-slave-7lbf krb5kdc[12220](info): commencing operation
krb5kdc: starting...
W20260805 22:47:34.745779 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:34 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:47:35.014297 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:35603
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:35603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:35.369669 12240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:35.370042 12240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:35.370211 12240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:35.377866 12240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:35.378026 12240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:35.378108 12240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:35.378185 12240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:35.378249 12240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:35.388780 12240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:35603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:35603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:35.390424 12240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:35.391909 12240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:35.404642 12249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:35.404568 12251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:35.404603 12253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:35.405432 12240 server_base.cc:1034] running on GCE node
I20260805 22:47:35.406356 12240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:35.407982 12240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:35.409225 12240 hybrid_clock.cc:648] HybridClock initialized: now 1785970055409167 us; error 45 us; skew 500 ppm
I20260805 22:47:35.409682 12240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:35 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970055, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:35.412981 12240 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:47:35.414681 12240 webserver.cc:460] Webserver started at http://127.4.28.62:42573/ using document root <none> and password file <none>
I20260805 22:47:35.415472 12240 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:35.415613 12240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:35.415951 12240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:35.419346 12240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "bd11efb05f8d47cc93f5718c53733580"
format_stamp: "Formatted at 2026-08-05 22:47:35 on dist-test-slave-7lbf"
server_key: "ef3dbb269ac1713ab01c5a182de62bf6"
server_key_iv: "41f2d6a4033f968d496ac4fe5463080c"
server_key_version: "encryptionkey@0"
I20260805 22:47:35.420209 12240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "bd11efb05f8d47cc93f5718c53733580"
format_stamp: "Formatted at 2026-08-05 22:47:35 on dist-test-slave-7lbf"
server_key: "ef3dbb269ac1713ab01c5a182de62bf6"
server_key_iv: "41f2d6a4033f968d496ac4fe5463080c"
server_key_version: "encryptionkey@0"
I20260805 22:47:35.426190 12240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260805 22:47:35.430316 12263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:35.431838 12240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:35.432024 12240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "bd11efb05f8d47cc93f5718c53733580"
format_stamp: "Formatted at 2026-08-05 22:47:35 on dist-test-slave-7lbf"
server_key: "ef3dbb269ac1713ab01c5a182de62bf6"
server_key_iv: "41f2d6a4033f968d496ac4fe5463080c"
server_key_version: "encryptionkey@0"
I20260805 22:47:35.432229 12240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:35.454830 12240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:35.459506 12240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:35.459816 12240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:35.478725 12240 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:35603
I20260805 22:47:35.478755 12336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:35603 every 8 connection(s)
I20260805 22:47:35.480289 12240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:35.482241 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 12240
I20260805 22:47:35.482502 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:35.483351 4208 external_mini_cluster.cc:1428] Setting key c517910cb0eb5b109a36703207cc01dc
I20260805 22:47:35.485549 12337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:47:35 dist-test-slave-7lbf krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:35.499656 12337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580: Bootstrap starting.
I20260805 22:47:35.503234 12337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:35.504559 12337 log.cc:824] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:35.508325 12337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580: No bootstrap required, opened a new log
I20260805 22:47:35.508414 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:35.485548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.4.28.62:35603)
0805 22:47:35.486433 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:35.486446 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:35.486488 (+ 42us) server_negotiation.cc:407] Connection header received
0805 22:47:35.488183 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:35.488208 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:35.488650 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:35.489010 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:35.490852 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:35.492262 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:35.493993 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:35.494718 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:35.494863 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:35.500166 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:35.500175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:35.500195 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:35.500242 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:35.503381 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:35.504297 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:35.504304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:35.504312 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:35.504523 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:35.504958 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:35.504962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:35.504964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:35.505637 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:35.505901 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:35.506473 (+ 572us) server_negotiation.cc:299] Negotiation successful
0805 22:47:35.506966 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":399,"threads_started":1}
I20260805 22:47:35.516902 12337 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11efb05f8d47cc93f5718c53733580" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35603 } }
I20260805 22:47:35.517333 12337 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:35.517441 12337 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd11efb05f8d47cc93f5718c53733580, State: Initialized, Role: FOLLOWER
I20260805 22:47:35.518206 12337 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11efb05f8d47cc93f5718c53733580" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35603 } }
I20260805 22:47:35.518529 12337 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:35.518697 12337 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:35.518878 12337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:35.521155 12337 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11efb05f8d47cc93f5718c53733580" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35603 } }
I20260805 22:47:35.521768 12337 leader_election.cc:302] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd11efb05f8d47cc93f5718c53733580; no voters:
I20260805 22:47:35.522477 12337 leader_election.cc:288] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:35.522615 12342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:35.523268 12342 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [term 1 LEADER]: Becoming Leader. State: Replica: bd11efb05f8d47cc93f5718c53733580, State: Running, Role: LEADER
I20260805 22:47:35.523839 12342 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11efb05f8d47cc93f5718c53733580" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35603 } }
I20260805 22:47:35.524688 12337 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:35.526491 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd11efb05f8d47cc93f5718c53733580. Latest consensus state: current_term: 1 leader_uuid: "bd11efb05f8d47cc93f5718c53733580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11efb05f8d47cc93f5718c53733580" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35603 } } }
I20260805 22:47:35.527066 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:35.527449 12343 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd11efb05f8d47cc93f5718c53733580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd11efb05f8d47cc93f5718c53733580" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35603 } } }
I20260805 22:47:35.527876 12343 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:35.529152 12348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:35.534768 12348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:35.544849 12348 catalog_manager.cc:1347] Generated new cluster ID: 1e514a874b4742dcb2dbe771e2b3fbca
I20260805 22:47:35.544989 12348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:35.575002 12348 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:35.576453 12348 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:35.583393 12348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd11efb05f8d47cc93f5718c53733580: Generated new TSK 0
I20260805 22:47:35.584607 12348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:47:35.658367 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:35603
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:36.017931 12368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:36.018276 12368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:36.018419 12368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:36.026266 12368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:36.026424 12368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:36.026605 12368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:36.037428 12368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:35603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:36.039017 12368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:36.040473 12368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:36.054803 12377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:36.054731 12378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:36.054765 12380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:36.056185 12368 server_base.cc:1034] running on GCE node
I20260805 22:47:36.056986 12368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:36.058213 12368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:36.059494 12368 hybrid_clock.cc:648] HybridClock initialized: now 1785970056059437 us; error 50 us; skew 500 ppm
I20260805 22:47:36.059880 12368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:36 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970056, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:36.063431 12368 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:47:36.065145 12368 webserver.cc:460] Webserver started at http://127.4.28.1:44445/ using document root <none> and password file <none>
I20260805 22:47:36.065984 12368 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:36.066123 12368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:36.066490 12368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:36.069737 12368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "7a52a46b64ef44bb863e025847816354"
format_stamp: "Formatted at 2026-08-05 22:47:36 on dist-test-slave-7lbf"
server_key: "ede2244cd211e232cb8a110439923ce2"
server_key_iv: "7054a4e0f9cec7265b03b40cb0f44709"
server_key_version: "encryptionkey@0"
I20260805 22:47:36.070645 12368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "7a52a46b64ef44bb863e025847816354"
format_stamp: "Formatted at 2026-08-05 22:47:36 on dist-test-slave-7lbf"
server_key: "ede2244cd211e232cb8a110439923ce2"
server_key_iv: "7054a4e0f9cec7265b03b40cb0f44709"
server_key_version: "encryptionkey@0"
I20260805 22:47:36.077186 12368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260805 22:47:36.082299 12390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:36.083999 12368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:36.084189 12368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "7a52a46b64ef44bb863e025847816354"
format_stamp: "Formatted at 2026-08-05 22:47:36 on dist-test-slave-7lbf"
server_key: "ede2244cd211e232cb8a110439923ce2"
server_key_iv: "7054a4e0f9cec7265b03b40cb0f44709"
server_key_version: "encryptionkey@0"
I20260805 22:47:36.084436 12368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:36.115329 12368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:36.120416 12368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:36.120751 12368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:36.121889 12368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:36.123459 12368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:36.123541 12368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:36.123649 12368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:36.123721 12368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:36.155784 12368 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:36825
I20260805 22:47:36.155836 12544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:36825 every 8 connection(s)
I20260805 22:47:36.157181 12368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:36.158360 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 12368
I20260805 22:47:36.158598 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:36.159231 4208 external_mini_cluster.cc:1428] Setting key c7c80e66f83bc818e1a03b2e13b816c8
Aug 05 22:47:36 dist-test-slave-7lbf krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970056, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:36.180051 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:36.160223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:46561 (local address 127.4.28.62:35603)
0805 22:47:36.160721 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:36.160727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:36.162054 (+ 1327us) server_negotiation.cc:407] Connection header received
0805 22:47:36.163927 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:36.163933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:36.164030 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:36.164128 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:36.165868 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:36.166611 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:36.168294 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:36.168706 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:36.168829 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:36.172762 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:36.172770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:36.172775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:36.172822 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:36.176287 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:36.177336 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:36.177343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:36.177347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:36.177600 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:36.178114 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:36.178119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:36.178122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:36.178464 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:36.178630 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:36.179453 (+ 823us) server_negotiation.cc:299] Negotiation successful
0805 22:47:36.179706 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
I20260805 22:47:36.180961 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:36.160907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:35603 (local address 127.4.28.1:46561)
0805 22:47:36.161853 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:36.161903 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:36.163627 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:36.164414 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:36.164423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:36.164988 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:36.165676 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:36.165700 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:36.166794 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:36.166798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:36.168116 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:36.168133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:36.168968 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:36.168976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:36.169225 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:36.169921 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:36.169945 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:36.172455 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:36.176526 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:36.176543 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:36.176564 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:36.177153 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:36.177717 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:36.177721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:36.177723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:36.177954 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:36.178616 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:36.178623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:36.179041 (+ 418us) client_negotiation.cc:769] Sending connection context
0805 22:47:36.179291 (+ 250us) client_negotiation.cc:240] Negotiation successful
0805 22:47:36.179740 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":420,"threads_started":1}
I20260805 22:47:36.182706 12545 heartbeater.cc:347] Connected to a master server at 127.4.28.62:35603
I20260805 22:47:36.183032 12545 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:36.183827 12545 heartbeater.cc:511] Master 127.4.28.62:35603 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
I20260805 22:47:36.186246 12287 ts_manager.cc:194] Registered new tserver with Master: 7a52a46b64ef44bb863e025847816354 (127.4.28.1:36825)
I20260805 22:47:36.187845 12287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:46561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:47:36.221691 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:35603
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:36.575248 12552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:36.575579 12552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:36.575677 12552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:36.583348 12552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:36.583482 12552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:36.583631 12552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:36.594524 12552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:35603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:36.595988 12552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:36.597458 12552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:36.611799 12560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:36.611837 12564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:36.611769 12559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:36.612855 12552 server_base.cc:1034] running on GCE node
I20260805 22:47:36.613677 12552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:36.614907 12552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:36.616174 12552 hybrid_clock.cc:648] HybridClock initialized: now 1785970056616113 us; error 56 us; skew 500 ppm
I20260805 22:47:36.616550 12552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:36 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970056, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:36.620064 12552 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:47:36.621778 12552 webserver.cc:460] Webserver started at http://127.4.28.2:40503/ using document root <none> and password file <none>
I20260805 22:47:36.622542 12552 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:36.622673 12552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:36.623013 12552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:36.626368 12552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "3b7b672bbef841b7a1838f50013e26d3"
format_stamp: "Formatted at 2026-08-05 22:47:36 on dist-test-slave-7lbf"
server_key: "dcbdd74a81bee9307acd7f48cad33bff"
server_key_iv: "61bb509bc1ebc7692b113fb7da4ddff1"
server_key_version: "encryptionkey@0"
I20260805 22:47:36.627218 12552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "3b7b672bbef841b7a1838f50013e26d3"
format_stamp: "Formatted at 2026-08-05 22:47:36 on dist-test-slave-7lbf"
server_key: "dcbdd74a81bee9307acd7f48cad33bff"
server_key_iv: "61bb509bc1ebc7692b113fb7da4ddff1"
server_key_version: "encryptionkey@0"
I20260805 22:47:36.633076 12552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:47:36.637004 12576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:36.638589 12552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:36.638778 12552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "3b7b672bbef841b7a1838f50013e26d3"
format_stamp: "Formatted at 2026-08-05 22:47:36 on dist-test-slave-7lbf"
server_key: "dcbdd74a81bee9307acd7f48cad33bff"
server_key_iv: "61bb509bc1ebc7692b113fb7da4ddff1"
server_key_version: "encryptionkey@0"
I20260805 22:47:36.639008 12552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:36.682842 12552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:36.687654 12552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:36.687965 12552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:36.689081 12552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:36.690656 12552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:36.690742 12552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:36.690869 12552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:36.690924 12552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:36.723183 12552 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:39893
I20260805 22:47:36.723254 12735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:39893 every 8 connection(s)
I20260805 22:47:36.724555 12552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:36.730449 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 12552
I20260805 22:47:36.730671 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:47:36.731241 4208 external_mini_cluster.cc:1428] Setting key f697fd60ab94c31a50e75562e0f911d5
Aug 05 22:47:36 dist-test-slave-7lbf krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970056, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:36.750900 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:36.728134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:35603 (local address 127.4.28.2:38585)
0805 22:47:36.728902 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:36.728948 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:36.730806 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:36.731649 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:36.731659 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:36.732305 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:36.733194 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:36.733221 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:36.734430 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:36.734435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:36.736390 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:36.736408 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:36.737327 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:36.737334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:36.737655 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:36.738410 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:36.738442 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:36.741097 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:36.745006 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:36.745017 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:36.745039 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:36.745791 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:36.746321 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:36.746325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:36.746329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:36.746621 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:36.747508 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:36.747518 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:36.748065 (+ 547us) client_negotiation.cc:769] Sending connection context
0805 22:47:36.748444 (+ 379us) client_negotiation.cc:240] Negotiation successful
0805 22:47:36.749112 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":315,"threads_started":1}
I20260805 22:47:36.751004 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:36.727479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:38585 (local address 127.4.28.62:35603)
0805 22:47:36.727956 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:36.727961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:36.729086 (+ 1125us) server_negotiation.cc:407] Connection header received
0805 22:47:36.731107 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:36.731113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:36.731209 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:36.731362 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:36.733395 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:36.734244 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:36.736548 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:36.737076 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:36.737175 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:36.741403 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:36.741411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:36.741415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:36.741463 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:36.744781 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:36.745962 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:36.745968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:36.745972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:36.746174 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:36.746781 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:36.746787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:36.746790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:36.747232 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:36.747408 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:36.749874 (+ 2466us) server_negotiation.cc:299] Negotiation successful
0805 22:47:36.750298 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":238,"threads_started":1}
I20260805 22:47:36.752985 12737 heartbeater.cc:347] Connected to a master server at 127.4.28.62:35603
I20260805 22:47:36.753471 12737 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:36.754743 12737 heartbeater.cc:511] Master 127.4.28.62:35603 requested a full tablet report, sending...
I20260805 22:47:36.757025 12287 ts_manager.cc:194] Registered new tserver with Master: 3b7b672bbef841b7a1838f50013e26d3 (127.4.28.2:39893)
I20260805 22:47:36.758104 12287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:38585
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:47:36.795593 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:35603
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:37.166422 12745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:37.166754 12745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:37.166896 12745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:37.174751 12745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:37.174898 12745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:37.175073 12745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:37.185781 12745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:35603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:37.187255 12745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:37.188776 12745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 22:47:37.191454 12545 heartbeater.cc:503] Master 127.4.28.62:35603 was elected leader, sending a full tablet report...
W20260805 22:47:37.204051 12754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:37.204053 12758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:37.204125 12753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:37.205202 12745 server_base.cc:1034] running on GCE node
I20260805 22:47:37.206033 12745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:37.207206 12745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:37.208482 12745 hybrid_clock.cc:648] HybridClock initialized: now 1785970057208425 us; error 62 us; skew 500 ppm
I20260805 22:47:37.208875 12745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:37.212684 12745 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:47:37.214766 12745 webserver.cc:460] Webserver started at http://127.4.28.3:41269/ using document root <none> and password file <none>
I20260805 22:47:37.215590 12745 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:37.215731 12745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:37.216094 12745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:37.219511 12745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "ff7b76c74cc34530aa386609a4f89ab4"
format_stamp: "Formatted at 2026-08-05 22:47:37 on dist-test-slave-7lbf"
server_key: "ee927e8eb3fa010c65ad6fb6ec801727"
server_key_iv: "2c7f45d36e4ab7e37ef221dd36e57c02"
server_key_version: "encryptionkey@0"
I20260805 22:47:37.220489 12745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "ff7b76c74cc34530aa386609a4f89ab4"
format_stamp: "Formatted at 2026-08-05 22:47:37 on dist-test-slave-7lbf"
server_key: "ee927e8eb3fa010c65ad6fb6ec801727"
server_key_iv: "2c7f45d36e4ab7e37ef221dd36e57c02"
server_key_version: "encryptionkey@0"
I20260805 22:47:37.226909 12745 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260805 22:47:37.230880 12770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:37.232439 12745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:37.232623 12745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "ff7b76c74cc34530aa386609a4f89ab4"
format_stamp: "Formatted at 2026-08-05 22:47:37 on dist-test-slave-7lbf"
server_key: "ee927e8eb3fa010c65ad6fb6ec801727"
server_key_iv: "2c7f45d36e4ab7e37ef221dd36e57c02"
server_key_version: "encryptionkey@0"
I20260805 22:47:37.232839 12745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:37.260236 12745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:37.264992 12745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:37.265311 12745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:37.266438 12745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:37.267997 12745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:37.268081 12745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:37.268208 12745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:37.268263 12745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:37.301007 12745 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:36049
I20260805 22:47:37.301047 12935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:36049 every 8 connection(s)
I20260805 22:47:37.302388 12745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:37.307093 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 12745
I20260805 22:47:37.307276 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:47:37.307713 4208 external_mini_cluster.cc:1428] Setting key c4b854a499d02b264f87459cc6aa3d0d
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:37.323117 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:37.305092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:35105 (local address 127.4.28.62:35603)
0805 22:47:37.305576 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.305581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.306668 (+ 1087us) server_negotiation.cc:407] Connection header received
0805 22:47:37.308745 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.308751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.308839 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.308943 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.310630 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.311374 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.312984 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.313403 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.313545 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:37.317064 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:37.317070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:37.317074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:37.317111 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:37.319709 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.320553 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.320558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.320561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.320704 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.321167 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.321171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.321173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.321465 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:37.321656 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:37.322418 (+ 762us) server_negotiation.cc:299] Negotiation successful
0805 22:47:37.322691 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":206,"threads_started":1}
I20260805 22:47:37.324152 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:37.305685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:35603 (local address 127.4.28.3:35105)
0805 22:47:37.306488 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:37.306538 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:37.308454 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:37.309215 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:37.309224 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:37.309831 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:37.310444 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:37.310462 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.311535 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.311539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:37.312828 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:37.312841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.313669 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.313677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:37.313916 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:37.314543 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:37.314567 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:37.316811 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:37.319896 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:37.319905 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:37.319922 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:37.320410 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:37.320821 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:37.320825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:37.320827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:37.321042 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:37.321620 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:37.321629 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:37.322015 (+ 386us) client_negotiation.cc:769] Sending connection context
0805 22:47:37.322277 (+ 262us) client_negotiation.cc:240] Negotiation successful
0805 22:47:37.322890 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":351,"threads_started":1}
I20260805 22:47:37.325620 12936 heartbeater.cc:347] Connected to a master server at 127.4.28.62:35603
I20260805 22:47:37.325965 12936 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:37.326706 12936 heartbeater.cc:511] Master 127.4.28.62:35603 requested a full tablet report, sending...
I20260805 22:47:37.328315 12287 ts_manager.cc:194] Registered new tserver with Master: ff7b76c74cc34530aa386609a4f89ab4 (127.4.28.3:36049)
I20260805 22:47:37.329186 12287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:35105
I20260805 22:47:37.336275 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/test-admin.keytab.
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:37.372352 4208 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260805 22:47:37.378890 12950 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:37.375521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.4.28.1:36825)
0805 22:47:37.376754 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.376763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.376793 (+ 30us) server_negotiation.cc:407] Connection header received
0805 22:47:37.376860 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.376868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.377124 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.377268 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.378442 (+ 1174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43344: BlockingRecv error: recv got EOF from 127.0.0.1:43344 (error 108)
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":886,"threads_started":1}
W20260805 22:47:37.379377 4208 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5cc)
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:37.381057 4208 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:37.382766 4208 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:47:37.401175 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:37.384804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.4.28.1:36825)
0805 22:47:37.385032 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.385038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.385258 (+ 220us) server_negotiation.cc:407] Connection header received
0805 22:47:37.385436 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.385440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.385510 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.385637 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.386732 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.387782 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.389229 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.389988 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.390125 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:37.394396 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:37.394415 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:37.394436 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:37.394492 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:37.398001 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.398847 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.398857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.398859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.399021 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.399535 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.399541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.399543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.399990 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:37.400252 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:37.400649 (+ 397us) server_negotiation.cc:299] Negotiation successful
0805 22:47:37.400883 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260805 22:47:37.407785 12950 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:37.405739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.4.28.1:36825)
0805 22:47:37.405957 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.405963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.406155 (+ 192us) server_negotiation.cc:407] Connection header received
0805 22:47:37.406340 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.406345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.406420 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.406500 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.407520 (+ 1020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43356: BlockingRecv error: recv got EOF from 127.0.0.1:43356 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260805 22:47:37.407948 4208 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5cc' of type FILE
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:37.409580 4208 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:37.410941 4208 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:47:37.426584 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:37.412491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.4.28.1:36825)
0805 22:47:37.412719 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.412726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.412866 (+ 140us) server_negotiation.cc:407] Connection header received
0805 22:47:37.413070 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.413074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.413144 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.413227 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.414444 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.415219 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.416501 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.416936 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.417067 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:37.421017 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:37.421024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:37.421027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:37.421086 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:37.423760 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.424576 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.424583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.424586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.424753 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.425158 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.425162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.425164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.425467 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:37.425774 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:37.426075 (+ 301us) server_negotiation.cc:299] Negotiation successful
0805 22:47:37.426271 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260805 22:47:37.431381 12950 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:37.429354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.4.28.1:36825)
0805 22:47:37.429611 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.429617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.429861 (+ 244us) server_negotiation.cc:407] Connection header received
0805 22:47:37.430048 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.430052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.430164 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.430247 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.431129 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43376: BlockingRecv error: recv got EOF from 127.0.0.1:43376 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260805 22:47:37.431588 4208 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785970002413767-4208-0/krb5kdc/krb5cc' of type FILE
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:37.433097 4208 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:37.434511 4208 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:47:37.450184 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:37.436118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.4.28.1:36825)
0805 22:47:37.436362 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.436368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.436677 (+ 309us) server_negotiation.cc:407] Connection header received
0805 22:47:37.436907 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.436912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.436986 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.437092 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.438187 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.438950 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.440237 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.440722 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.440856 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:37.444722 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:37.444729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:37.444733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:37.444782 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:37.447472 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.448237 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.448241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.448244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.448402 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.448784 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.448789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.448791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.449117 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:37.449388 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:37.449689 (+ 301us) server_negotiation.cc:299] Negotiation successful
0805 22:47:37.449912 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260805 22:47:37.451998 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 12368
I20260805 22:47:37.465669 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:37.453019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.4.28.1:36825)
0805 22:47:37.453265 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.453272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.453723 (+ 451us) server_negotiation.cc:407] Connection header received
0805 22:47:37.453842 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.453847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.453919 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.454011 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.454954 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.455668 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.457020 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.457509 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.457693 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:37.460207 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:37.460213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:37.460216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:37.460256 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:37.462725 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.463447 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.463451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.463453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.463601 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.464049 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.464053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.464055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.464375 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:37.464591 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:37.465041 (+ 450us) server_negotiation.cc:299] Negotiation successful
0805 22:47:37.465288 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260805 22:47:37.466426 12539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:37.550631 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 12368
I20260805 22:47:37.577907 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 12552
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:47:37.594270 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:37.578853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.4.28.2:39893)
0805 22:47:37.579451 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.579459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.579496 (+ 37us) server_negotiation.cc:407] Connection header received
0805 22:47:37.579561 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.579567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.579780 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.579920 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.580957 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.582128 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.583449 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.584067 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.584179 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:37.588372 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:37.588388 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:37.588406 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:37.588451 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:37.591255 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.592003 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.592011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.592013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.592148 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.592524 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.592529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.592531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.592948 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:37.593245 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:37.593709 (+ 464us) server_negotiation.cc:299] Negotiation successful
0805 22:47:37.593948 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":291,"threads_started":1}
I20260805 22:47:37.595705 12728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:37.678785 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 12552
I20260805 22:47:37.697393 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 12745
Aug 05 22:47:37 dist-test-slave-7lbf krb5kdc[12220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:47:37.713328 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:37.698568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.4.28.3:36049)
0805 22:47:37.699231 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:37.699238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:37.699269 (+ 31us) server_negotiation.cc:407] Connection header received
0805 22:47:37.699334 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:37.699340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:37.699574 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:37.699702 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:37.700789 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.701954 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.703291 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:37.703883 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:37.703991 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:37.708003 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:37.708017 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:37.708036 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:37.708076 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:37.710824 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.711537 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.711545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.711547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.711667 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:37.711972 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:37.711974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:37.711976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:37.712319 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:37.712570 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:37.712836 (+ 266us) server_negotiation.cc:299] Negotiation successful
0805 22:47:37.713044 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":353,"threads_started":1}
I20260805 22:47:37.714740 12931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:37.798676 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 12745
I20260805 22:47:37.818449 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 12240
I20260805 22:47:37.819744 12332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:37.925019 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 12240
2026-08-05T22:47:37Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4399 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:38 dist-test-slave-7lbf krb5kdc[12981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:38 dist-test-slave-7lbf krb5kdc[12981](info): set up 2 sockets
Aug 05 22:47:38 dist-test-slave-7lbf krb5kdc[12981](info): commencing operation
krb5kdc: starting...
W20260805 22:47:39.147512 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:39 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.62.keytab.
Entry for principal oryx/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.62.keytab.
I20260805 22:47:39.422367 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45685
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.62.keytab
--principal=oryx/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:39.773435 13000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:39.773837 13000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:39.773989 13000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:39.781595 13000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:39.781739 13000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:39.781814 13000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:39.781904 13000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:39.781966 13000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:39.792512 13000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.62.keytab
--principal=oryx/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:39.794000 13000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:39.795398 13000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:39.807525 13008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:39.807485 13011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:39.808204 13000 server_base.cc:1034] running on GCE node
W20260805 22:47:39.808277 13007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:39.809149 13000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:39.810791 13000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:39.812031 13000 hybrid_clock.cc:648] HybridClock initialized: now 1785970059811980 us; error 37 us; skew 500 ppm
I20260805 22:47:39.812415 13000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:39 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:39.819169 13000 init.cc:377] Logged in from keytab as oryx/127.4.28.62@KRBTEST.COM (short username oryx)
I20260805 22:47:39.820919 13000 webserver.cc:460] Webserver started at http://127.4.28.62:37159/ using document root <none> and password file <none>
I20260805 22:47:39.821730 13000 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:39.821869 13000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:39.822194 13000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:39.825434 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "87fc4f6a5854435f93c862b049b98b9e"
format_stamp: "Formatted at 2026-08-05 22:47:39 on dist-test-slave-7lbf"
server_key: "3b065aece6780cba8715d660085bcda0"
server_key_iv: "97b2ea277cd6b7881271f56535b37c8e"
server_key_version: "encryptionkey@0"
I20260805 22:47:39.826299 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "87fc4f6a5854435f93c862b049b98b9e"
format_stamp: "Formatted at 2026-08-05 22:47:39 on dist-test-slave-7lbf"
server_key: "3b065aece6780cba8715d660085bcda0"
server_key_iv: "97b2ea277cd6b7881271f56535b37c8e"
server_key_version: "encryptionkey@0"
I20260805 22:47:39.832062 13000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260805 22:47:39.835920 13019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:39.837450 13000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:47:39.837692 13000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "87fc4f6a5854435f93c862b049b98b9e"
format_stamp: "Formatted at 2026-08-05 22:47:39 on dist-test-slave-7lbf"
server_key: "3b065aece6780cba8715d660085bcda0"
server_key_iv: "97b2ea277cd6b7881271f56535b37c8e"
server_key_version: "encryptionkey@0"
I20260805 22:47:39.837939 13000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:39.873185 13000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:39.877945 13000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:39.878259 13000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:39.897632 13000 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:45685
I20260805 22:47:39.897647 13090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:45685 every 8 connection(s)
I20260805 22:47:39.899169 13000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:39.900264 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 13000
I20260805 22:47:39.900511 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:39.901180 4208 external_mini_cluster.cc:1428] Setting key 112c70c6cc522690ad3ffc4a2271e78a
I20260805 22:47:39.903904 13092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:47:39 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:39.917974 13092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: Bootstrap starting.
I20260805 22:47:39.922346 13092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:39.923682 13092 log.cc:824] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:39.927632 13092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: No bootstrap required, opened a new log
I20260805 22:47:39.927714 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:39.903498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.4.28.62:45685)
0805 22:47:39.904396 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:39.904413 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:39.904472 (+ 59us) server_negotiation.cc:407] Connection header received
0805 22:47:39.906821 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:39.906870 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:39.907443 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:39.907961 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:39.909448 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:39.911618 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:39.913046 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:39.914123 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:39.914353 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:39.918597 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:39.918609 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:39.918630 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:39.918683 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:39.922651 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:39.923467 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:39.923478 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:39.923488 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:39.923724 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:39.924165 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:39.924170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:39.924173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:39.924894 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:39.925147 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:39.925794 (+ 647us) server_negotiation.cc:299] Negotiation successful
0805 22:47:39.926319 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":371,"threads_started":1}
I20260805 22:47:39.936721 13092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } }
I20260805 22:47:39.937134 13092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:39.937222 13092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87fc4f6a5854435f93c862b049b98b9e, State: Initialized, Role: FOLLOWER
I20260805 22:47:39.937973 13092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } }
I20260805 22:47:39.938292 13092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:39.938460 13092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:39.938628 13092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:39.940838 13092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } }
I20260805 22:47:39.941341 13092 leader_election.cc:302] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87fc4f6a5854435f93c862b049b98b9e; no voters:
I20260805 22:47:39.942015 13092 leader_election.cc:288] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:39.942178 13098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:39.942931 13098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 1 LEADER]: Becoming Leader. State: Replica: 87fc4f6a5854435f93c862b049b98b9e, State: Running, Role: LEADER
I20260805 22:47:39.943485 13098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } }
I20260805 22:47:39.944296 13092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:39.946467 13101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87fc4f6a5854435f93c862b049b98b9e. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } }
I20260805 22:47:39.946916 13101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:39.947467 13100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } }
I20260805 22:47:39.947827 13100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:39.948402 13108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:39.953377 13108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:39.963728 13108 catalog_manager.cc:1347] Generated new cluster ID: 709c639440b74ebf857ee564da01e9ad
I20260805 22:47:39.963850 13108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:39.997869 13108 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:39.999385 13108 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:40.010629 13108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: Generated new TSK 0
I20260805 22:47:40.011631 13108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab.
Entry for principal oryx/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab.
I20260805 22:47:40.094697 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45685
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab
--principal=oryx/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:40.481077 13130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:40.481434 13130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:40.481626 13130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:40.489169 13130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:40.489331 13130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:40.489494 13130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:40.499893 13130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab
--principal=oryx/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:40.501390 13130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:40.502826 13130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:40.516899 13135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:40.516942 13139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:40.516858 13137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:40.517629 13130 server_base.cc:1034] running on GCE node
I20260805 22:47:40.518277 13130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:40.519284 13130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:40.520502 13130 hybrid_clock.cc:648] HybridClock initialized: now 1785970060520452 us; error 32 us; skew 500 ppm
I20260805 22:47:40.520828 13130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:40 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970060, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:40.524217 13130 init.cc:377] Logged in from keytab as oryx/127.4.28.1@KRBTEST.COM (short username oryx)
I20260805 22:47:40.525805 13130 webserver.cc:460] Webserver started at http://127.4.28.1:36929/ using document root <none> and password file <none>
I20260805 22:47:40.526554 13130 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:40.526686 13130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:40.527012 13130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:40.530202 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "2e0b768323044f0b8a357b08ac2054ed"
format_stamp: "Formatted at 2026-08-05 22:47:40 on dist-test-slave-7lbf"
server_key: "5d20f4b16717e718f08ad836b6856c95"
server_key_iv: "521273a1233adcf84b40e994460e562c"
server_key_version: "encryptionkey@0"
I20260805 22:47:40.531049 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "2e0b768323044f0b8a357b08ac2054ed"
format_stamp: "Formatted at 2026-08-05 22:47:40 on dist-test-slave-7lbf"
server_key: "5d20f4b16717e718f08ad836b6856c95"
server_key_iv: "521273a1233adcf84b40e994460e562c"
server_key_version: "encryptionkey@0"
I20260805 22:47:40.536679 13130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260805 22:47:40.540313 13150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:40.541817 13130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260805 22:47:40.541994 13130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "2e0b768323044f0b8a357b08ac2054ed"
format_stamp: "Formatted at 2026-08-05 22:47:40 on dist-test-slave-7lbf"
server_key: "5d20f4b16717e718f08ad836b6856c95"
server_key_iv: "521273a1233adcf84b40e994460e562c"
server_key_version: "encryptionkey@0"
I20260805 22:47:40.542199 13130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:40.573212 13130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:40.577932 13130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:40.578243 13130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:40.579280 13130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:40.580746 13130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:40.580833 13130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:40.580960 13130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:40.581032 13130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:40.614482 13130 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:33079
I20260805 22:47:40.614562 13300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:33079 every 8 connection(s)
I20260805 22:47:40.615967 13130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:40.617985 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 13130
I20260805 22:47:40.618208 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:40.618927 4208 external_mini_cluster.cc:1428] Setting key 770ade9b4d3dcd32daa0f21c9caf46bf
Aug 05 22:47:40 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970060, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:40.638263 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:40.618888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:50791 (local address 127.4.28.62:45685)
0805 22:47:40.619847 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:40.619851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:40.620898 (+ 1047us) server_negotiation.cc:407] Connection header received
0805 22:47:40.622804 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:40.622811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:40.622899 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:40.623016 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:40.624675 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:40.625664 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:40.627548 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:40.628088 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:40.628216 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:40.632186 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:40.632192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:40.632196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:40.632233 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:40.634882 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:40.635723 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:40.635729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:40.635731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:40.635878 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:40.636351 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:40.636355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:40.636357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:40.636658 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:40.636809 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:40.637635 (+ 826us) server_negotiation.cc:299] Negotiation successful
0805 22:47:40.637896 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":669,"threads_started":1}
I20260805 22:47:40.639420 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:40.619596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.1:50791)
0805 22:47:40.620560 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:40.620626 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:40.622498 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:40.623253 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:40.623261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:40.623827 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:40.624470 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:40.624492 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:40.625893 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:40.625898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:40.627353 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:40.627370 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:40.628358 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:40.628364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:40.628610 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:40.629318 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:40.629346 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:40.631911 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:40.635069 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:40.635080 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:40.635098 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:40.635584 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:40.635998 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:40.636003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:40.636007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:40.636228 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:40.636801 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:40.636811 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:40.637202 (+ 391us) client_negotiation.cc:769] Sending connection context
0805 22:47:40.637479 (+ 277us) client_negotiation.cc:240] Negotiation successful
0805 22:47:40.638104 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":389,"threads_started":1}
I20260805 22:47:40.641121 13301 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45685
I20260805 22:47:40.641456 13301 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:40.642310 13301 heartbeater.cc:511] Master 127.4.28.62:45685 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.4.28.2@KRBTEST.COM; defaulting to no policy
I20260805 22:47:40.644955 13048 ts_manager.cc:194] Registered new tserver with Master: 2e0b768323044f0b8a357b08ac2054ed (127.4.28.1:33079)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.2@KRBTEST.COM" created.
I20260805 22:47:40.646790 13048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:50791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab.
Entry for principal oryx/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab.
I20260805 22:47:40.677304 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45685
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab
--principal=oryx/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:41.035760 13309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:41.036104 13309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:41.036234 13309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:41.043802 13309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:41.043931 13309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:41.044080 13309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:41.054443 13309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab
--principal=oryx/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:41.055891 13309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:41.057293 13309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:41.071347 13316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:41.071385 13319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:41.071410 13315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:41.072233 13309 server_base.cc:1034] running on GCE node
I20260805 22:47:41.072865 13309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:41.073905 13309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:41.075114 13309 hybrid_clock.cc:648] HybridClock initialized: now 1785970061075053 us; error 49 us; skew 500 ppm
I20260805 22:47:41.075443 13309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:41 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970061, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:41.078860 13309 init.cc:377] Logged in from keytab as oryx/127.4.28.2@KRBTEST.COM (short username oryx)
I20260805 22:47:41.080417 13309 webserver.cc:460] Webserver started at http://127.4.28.2:32771/ using document root <none> and password file <none>
I20260805 22:47:41.081147 13309 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:41.081260 13309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:41.081562 13309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:41.084803 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "2761cac4c2f84655889032c0b97978e1"
format_stamp: "Formatted at 2026-08-05 22:47:41 on dist-test-slave-7lbf"
server_key: "441c4c5a3dd4ae357c29c16be4effda3"
server_key_iv: "6fe6582315278861a0d28b932d12bce7"
server_key_version: "encryptionkey@0"
I20260805 22:47:41.085688 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "2761cac4c2f84655889032c0b97978e1"
format_stamp: "Formatted at 2026-08-05 22:47:41 on dist-test-slave-7lbf"
server_key: "441c4c5a3dd4ae357c29c16be4effda3"
server_key_iv: "6fe6582315278861a0d28b932d12bce7"
server_key_version: "encryptionkey@0"
I20260805 22:47:41.091495 13309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260805 22:47:41.096676 13328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:41.098728 13309 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260805 22:47:41.098966 13309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "2761cac4c2f84655889032c0b97978e1"
format_stamp: "Formatted at 2026-08-05 22:47:41 on dist-test-slave-7lbf"
server_key: "441c4c5a3dd4ae357c29c16be4effda3"
server_key_iv: "6fe6582315278861a0d28b932d12bce7"
server_key_version: "encryptionkey@0"
I20260805 22:47:41.099208 13309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:41.131991 13309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:41.137120 13309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:41.137467 13309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:41.138693 13309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:41.140316 13309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:41.140436 13309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:41.140583 13309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:41.140661 13309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:41.175892 13309 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:42539
I20260805 22:47:41.175928 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:42539 every 8 connection(s)
I20260805 22:47:41.177433 13309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:41.185580 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 13309
I20260805 22:47:41.185781 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:47:41.186266 4208 external_mini_cluster.cc:1428] Setting key 6e36667017fe841f5603eb41cec5d789
Aug 05 22:47:41 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970061, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:41.199571 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:41.180394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:53175 (local address 127.4.28.62:45685)
0805 22:47:41.180965 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:41.180970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:41.182105 (+ 1135us) server_negotiation.cc:407] Connection header received
0805 22:47:41.184041 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:41.184047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:41.184139 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:41.184240 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:41.186007 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:41.186815 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:41.188716 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:41.189161 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:41.189276 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:41.193038 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:41.193044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:41.193048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:41.193087 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:41.195940 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:41.196863 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:41.196869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:41.196871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:41.197031 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:41.197569 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:41.197576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:41.197580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:41.197900 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:41.198053 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:41.198907 (+ 854us) server_negotiation.cc:299] Negotiation successful
0805 22:47:41.199172 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
I20260805 22:47:41.200663 13497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:41.180982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.2:53175)
0805 22:47:41.181903 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:41.181955 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:41.183724 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:41.184528 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:41.184537 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:41.185102 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:41.185783 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:41.185806 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:41.186980 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:41.186985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:41.188565 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:41.188579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:41.189399 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:41.189408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:41.189703 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:41.190398 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:41.190424 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:41.192738 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:41.196165 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:41.196174 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:41.196192 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:41.196700 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:41.197157 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:41.197161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:41.197164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:41.197398 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:41.198048 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:41.198058 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:41.198486 (+ 428us) client_negotiation.cc:769] Sending connection context
0805 22:47:41.198760 (+ 274us) client_negotiation.cc:240] Negotiation successful
0805 22:47:41.199364 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":342,"threads_started":1}
I20260805 22:47:41.202309 13494 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45685
I20260805 22:47:41.202718 13494 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:41.203728 13494 heartbeater.cc:511] Master 127.4.28.62:45685 requested a full tablet report, sending...
I20260805 22:47:41.205614 13048 ts_manager.cc:194] Registered new tserver with Master: 2761cac4c2f84655889032c0b97978e1 (127.4.28.2:42539)
I20260805 22:47:41.206504 13048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:53175
WARNING: no policy specified for oryx/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab.
Entry for principal oryx/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab.
I20260805 22:47:41.246878 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45685
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab
--principal=oryx/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:41.597931 13501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:41.598253 13501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:41.598356 13501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:41.605870 13501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:41.606000 13501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:41.606118 13501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:41.616379 13501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab
--principal=oryx/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:41.617796 13501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:41.619280 13501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:41.633226 13509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:41.633252 13508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:41.633172 13511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:41.634205 13501 server_base.cc:1034] running on GCE node
I20260805 22:47:41.634832 13501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:41.635816 13501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:41.637034 13501 hybrid_clock.cc:648] HybridClock initialized: now 1785970061636975 us; error 50 us; skew 500 ppm
I20260805 22:47:41.637342 13501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:41 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970061, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:41.640635 13501 init.cc:377] Logged in from keytab as oryx/127.4.28.3@KRBTEST.COM (short username oryx)
I20260805 22:47:41.642385 13501 webserver.cc:460] Webserver started at http://127.4.28.3:34723/ using document root <none> and password file <none>
I20260805 22:47:41.643172 13501 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:41.643293 13501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:41.643641 13501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:41.647158 13501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "79ed51ed787046eeba71558825437f70"
format_stamp: "Formatted at 2026-08-05 22:47:41 on dist-test-slave-7lbf"
server_key: "22f330dcf5e36e54448152a2797eee97"
server_key_iv: "8f0166f857b9824cb47ef1afeb0694ad"
server_key_version: "encryptionkey@0"
I20260805 22:47:41.648027 13501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "79ed51ed787046eeba71558825437f70"
format_stamp: "Formatted at 2026-08-05 22:47:41 on dist-test-slave-7lbf"
server_key: "22f330dcf5e36e54448152a2797eee97"
server_key_iv: "8f0166f857b9824cb47ef1afeb0694ad"
server_key_version: "encryptionkey@0"
I20260805 22:47:41.650100 13301 heartbeater.cc:503] Master 127.4.28.62:45685 was elected leader, sending a full tablet report...
I20260805 22:47:41.654147 13501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260805 22:47:41.658226 13522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:41.659866 13501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:41.660104 13501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "79ed51ed787046eeba71558825437f70"
format_stamp: "Formatted at 2026-08-05 22:47:41 on dist-test-slave-7lbf"
server_key: "22f330dcf5e36e54448152a2797eee97"
server_key_iv: "8f0166f857b9824cb47ef1afeb0694ad"
server_key_version: "encryptionkey@0"
I20260805 22:47:41.660331 13501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:41.680433 13501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:41.685760 13501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:41.686087 13501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:41.687180 13501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:41.688755 13501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:41.688853 13501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:41.688998 13501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:41.689056 13501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:41.721256 13501 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:40595
I20260805 22:47:41.721303 13678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:40595 every 8 connection(s)
I20260805 22:47:41.722631 13501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:41.725033 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 13501
I20260805 22:47:41.725255 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:47:41.725888 4208 external_mini_cluster.cc:1428] Setting key 08d91af6dfc9447e6eab78885354c4bd
Aug 05 22:47:41 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970061, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:41.744215 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:41.725432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:53021 (local address 127.4.28.62:45685)
0805 22:47:41.726093 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:41.726099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:41.727357 (+ 1258us) server_negotiation.cc:407] Connection header received
0805 22:47:41.728949 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:41.728955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:41.729044 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:41.729137 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:41.730773 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:41.731535 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:41.733120 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:41.733551 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:41.733671 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:41.737267 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:41.737273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:41.737277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:41.737318 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:41.740793 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:41.741856 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:41.741862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:41.741867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:41.742008 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:41.742434 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:41.742438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:41.742440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:41.742804 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:41.742935 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:41.743647 (+ 712us) server_negotiation.cc:299] Negotiation successful
0805 22:47:41.743843 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":273,"threads_started":1}
I20260805 22:47:41.745263 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:41.726076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.3:53021)
0805 22:47:41.726908 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:41.726966 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:41.728669 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:41.729406 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:41.729414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:41.729997 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:41.730599 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:41.730618 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:41.731702 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:41.731706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:41.732969 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:41.732984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:41.733790 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:41.733799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:41.734028 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:41.734689 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:41.734718 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:41.736971 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:41.741011 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:41.741023 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:41.741046 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:41.741722 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:41.742118 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:41.742121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:41.742123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:41.742332 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:41.742911 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:41.742918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:41.743291 (+ 373us) client_negotiation.cc:769] Sending connection context
0805 22:47:41.743551 (+ 260us) client_negotiation.cc:240] Negotiation successful
0805 22:47:41.744016 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":332,"threads_started":1}
I20260805 22:47:41.746852 13679 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45685
I20260805 22:47:41.747180 13679 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:41.747967 13679 heartbeater.cc:511] Master 127.4.28.62:45685 requested a full tablet report, sending...
I20260805 22:47:41.749619 13048 ts_manager.cc:194] Registered new tserver with Master: 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595)
I20260805 22:47:41.750484 13048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:53021
I20260805 22:47:41.754191 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.28.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.61.keytab.
Entry for principal oryx/127.4.28.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.61.keytab.
I20260805 22:47:41.817229 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.61:40937
--webserver_interface=127.4.28.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.61.keytab
--principal=oryx/127.4.28.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45685,127.4.28.61:40937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:42.171063 13687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:42.171386 13687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:42.171500 13687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:42.179140 13687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:42.179282 13687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:42.179363 13687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:42.179440 13687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:42.179502 13687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:42.189790 13687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45685,127.4.28.61:40937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.61.keytab
--principal=oryx/127.4.28.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.61:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.28.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
W20260805 22:47:42.190218 13687 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.28.62:45685,127.4.28.61:40937'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260805 22:47:42.191350 13687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:42.192731 13687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:42.205546 13692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:42.205489 13693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:42.205570 13696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:42.206278 13687 server_base.cc:1034] running on GCE node
I20260805 22:47:42.207244 13687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:42.208806 13687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:42.210054 13687 hybrid_clock.cc:648] HybridClock initialized: now 1785970062209994 us; error 56 us; skew 500 ppm
I20260805 22:47:42.210143 13494 heartbeater.cc:503] Master 127.4.28.62:45685 was elected leader, sending a full tablet report...
I20260805 22:47:42.210423 13687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:42 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970062, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:42.214107 13687 init.cc:377] Logged in from keytab as oryx/127.4.28.61@KRBTEST.COM (short username oryx)
I20260805 22:47:42.215667 13687 webserver.cc:460] Webserver started at http://127.4.28.61:34553/ using document root <none> and password file <none>
I20260805 22:47:42.216394 13687 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:42.216501 13687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:42.216825 13687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:42.220187 13687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data/instance:
uuid: "1a8ab594305c439ab3b78b51bbaec473"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "79adaebc42401a51bd2d94aa5d678c96"
server_key_iv: "6824317677b71076b77c189a1773170e"
server_key_version: "encryptionkey@0"
I20260805 22:47:42.221038 13687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal/instance:
uuid: "1a8ab594305c439ab3b78b51bbaec473"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "79adaebc42401a51bd2d94aa5d678c96"
server_key_iv: "6824317677b71076b77c189a1773170e"
server_key_version: "encryptionkey@0"
I20260805 22:47:42.227066 13687 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260805 22:47:42.231084 13704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:42.232754 13687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:42.232985 13687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal
uuid: "1a8ab594305c439ab3b78b51bbaec473"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "79adaebc42401a51bd2d94aa5d678c96"
server_key_iv: "6824317677b71076b77c189a1773170e"
server_key_version: "encryptionkey@0"
I20260805 22:47:42.233297 13687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:42.253386 13687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:42.258487 13687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:42.258797 13687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:42.279328 13687 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.61:40937
I20260805 22:47:42.279362 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.61:40937 every 8 connection(s)
I20260805 22:47:42.280861 13687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
I20260805 22:47:42.284583 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 13687
I20260805 22:47:42.284965 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:42.295496 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:47:42 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.61@KRBTEST.COM
Aug 05 22:47:42 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970062, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.61@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:42.311599 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.287106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.4.28.61:40937)
0805 22:47:42.287972 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.287993 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.288056 (+ 63us) server_negotiation.cc:407] Connection header received
0805 22:47:42.290147 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.290184 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.290859 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.291381 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:42.292868 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.294802 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.296263 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.297265 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.297451 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.302540 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:42.302553 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:42.302570 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:42.302634 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:42.306851 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.307678 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.307685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.307693 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.307868 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.308321 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.308327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.308330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.309035 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:42.309264 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:42.309965 (+ 701us) server_negotiation.cc:299] Negotiation successful
0805 22:47:42.310398 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":345,"threads_started":1}
I20260805 22:47:42.315063 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.298574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.4.28.62:45685)
0805 22:47:42.299124 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.299132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.299666 (+ 534us) server_negotiation.cc:407] Connection header received
0805 22:47:42.299861 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.299866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.299944 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.300037 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:42.301155 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.302256 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.304405 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.304836 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.304951 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.308864 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:42.308870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:42.308874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:42.308935 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:42.312184 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.312901 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.312906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.312908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.313042 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.313451 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.313455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.313457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.313828 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:42.313969 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:42.314487 (+ 518us) server_negotiation.cc:299] Negotiation successful
0805 22:47:42.314735 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":305,"threads_started":1}
I20260805 22:47:42.315099 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.298611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53310)
0805 22:47:42.299502 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:42.299554 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:42.299727 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:42.300135 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:42.300143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:42.300743 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:42.300996 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.301004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.302412 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.302418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.304233 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.304253 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.305043 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.305051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.305252 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.305929 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:42.305953 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:42.308622 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:42.312324 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.312336 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.312342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.312770 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.313155 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.313159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.313161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.313337 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.313967 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:42.313975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:42.314146 (+ 171us) client_negotiation.cc:769] Sending connection context
0805 22:47:42.314337 (+ 191us) client_negotiation.cc:240] Negotiation successful
0805 22:47:42.314681 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":491,"threads_started":1}
I20260805 22:47:42.316859 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:47:42 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970062, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.61@KRBTEST.COM for oryx/127.4.28.61@KRBTEST.COM
I20260805 22:47:42.329962 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.317471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.0.0.1:54392)
0805 22:47:42.317695 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:42.317714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:42.317809 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:42.318047 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:42.318050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:42.318362 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:42.318503 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.318511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.319442 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.319446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.320355 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.320367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.321049 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.321054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.321180 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.321762 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:42.321780 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:42.323924 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:42.327305 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.327312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.327317 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.327718 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.328103 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.328109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.328113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.328254 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.328856 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:42.328861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:42.329028 (+ 167us) client_negotiation.cc:769] Sending connection context
0805 22:47:42.329166 (+ 138us) client_negotiation.cc:240] Negotiation successful
0805 22:47:42.329486 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260805 22:47:42.330056 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.317677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.4.28.61:40937)
0805 22:47:42.317807 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.317811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.317827 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:47:42.317886 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.317889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.317949 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.318021 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:42.318615 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.319323 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.320499 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.320889 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.320998 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.324148 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:42.324153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:42.324156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:42.324188 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:42.327143 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.327849 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.327856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.327860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.327989 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.328367 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.328373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.328376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.328723 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:42.328873 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:42.329268 (+ 395us) server_negotiation.cc:299] Negotiation successful
0805 22:47:42.329699 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":43}
I20260805 22:47:42.335322 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: Bootstrap starting.
I20260805 22:47:42.338853 13783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:42.340022 13783 log.cc:824] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:42.343300 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: No bootstrap required, opened a new log
I20260805 22:47:42.350910 13783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } }
I20260805 22:47:42.351271 13783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:42.351377 13783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8ab594305c439ab3b78b51bbaec473, State: Initialized, Role: FOLLOWER
I20260805 22:47:42.352011 13783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } }
I20260805 22:47:42.352864 13791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } } }
I20260805 22:47:42.353169 13791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:47:42.353979 13783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: configured and running, proceeding with master startup.
W20260805 22:47:42.363094 13806 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 22:47:42.363265 13806 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 22:47:42.367317 13687 master_runner.cc:428] Detected that this master 1a8ab594305c439ab3b78b51bbaec473 is joining an existing cluster
I20260805 22:47:42.367439 13687 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.28.61:40937
I20260805 22:47:42.392437 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.372708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.4.28.61:40937)
0805 22:47:42.374910 (+ 2202us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.374916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.377224 (+ 2308us) server_negotiation.cc:407] Connection header received
0805 22:47:42.377399 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.377404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.377477 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.377579 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:42.379283 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.380157 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.381747 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.382525 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.382666 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.384998 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:42.385010 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:42.385015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:42.385054 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:42.388840 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.389896 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.389902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.389910 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.390095 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.390521 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.390526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.390530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.390972 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:42.391148 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:42.391751 (+ 603us) server_negotiation.cc:299] Negotiation successful
0805 22:47:42.392018 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":853,"spinlock_wait_cycles":44288}
I20260805 22:47:42.393623 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.373049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.4.28.62:45685)
0805 22:47:42.374771 (+ 1722us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.374778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.376830 (+ 2052us) server_negotiation.cc:407] Connection header received
0805 22:47:42.377407 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.377412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.377505 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.377593 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:42.379378 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.380504 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.383262 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.383929 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.384065 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.386892 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:42.386899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:42.386902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:42.386983 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:42.390446 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.391249 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.391255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.391258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.391419 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.391864 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.391869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.391873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.392318 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:42.392452 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:42.393032 (+ 580us) server_negotiation.cc:299] Negotiation successful
0805 22:47:42.393278 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1539}
I20260805 22:47:42.393878 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.372538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.0.0.1:54406)
0805 22:47:42.377125 (+ 4587us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:42.377142 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:42.377299 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:42.377801 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:42.377806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:42.378626 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:42.379112 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.379132 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.380336 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.380340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.381454 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.381471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.382739 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.382746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.382913 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.383625 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:42.383655 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:42.384728 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:42.389106 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.389111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.389115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.389762 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.390234 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.390239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.390242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.390406 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.391218 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:42.391222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:42.391384 (+ 162us) client_negotiation.cc:769] Sending connection context
0805 22:47:42.393050 (+ 1666us) client_negotiation.cc:240] Negotiation successful
0805 22:47:42.393393 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4454,"mutex_wait_us":1,"thread_start_us":4338,"threads_started":1}
I20260805 22:47:42.395967 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.372498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53320)
0805 22:47:42.376660 (+ 4162us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:42.376687 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:42.377217 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:42.377801 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:42.377806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:42.378692 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:42.379151 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.379168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.380744 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.380749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.383060 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.383078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.384375 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.384382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.384574 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.385439 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:42.385460 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:42.386646 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:42.390648 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.390653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.390656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.391096 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.391570 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.391575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.391578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.391737 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.392537 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:42.392541 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:42.392734 (+ 193us) client_negotiation.cc:769] Sending connection context
0805 22:47:42.394168 (+ 1434us) client_negotiation.cc:240] Negotiation successful
0805 22:47:42.394535 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4001,"mutex_wait_us":27,"thread_start_us":3796,"threads_started":1}
WARNING: no policy specified for oryx/127.4.28.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.60@KRBTEST.COM" created.
Aug 05 22:47:42 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.62@KRBTEST.COM for oryx/127.4.28.61@KRBTEST.COM
I20260805 22:47:42.425304 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.404585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.4.28.61:40937)
0805 22:47:42.404798 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.404806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.405600 (+ 794us) server_negotiation.cc:407] Connection header received
0805 22:47:42.406098 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.406105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.406229 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.406314 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:42.408293 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.409414 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.411612 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.412360 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.412470 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.417362 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:42.417370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:42.417375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:42.417431 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:42.421319 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.422587 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.422594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.422598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.422777 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.423306 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.423316 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.423320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.423754 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:42.423893 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:42.424619 (+ 726us) server_negotiation.cc:299] Negotiation successful
0805 22:47:42.424907 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:47:42.426503 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.404585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.0.0.1:54422)
0805 22:47:42.405427 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:42.405476 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:42.405805 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:42.406683 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:42.406692 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:42.407738 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:42.408084 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.408101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.409669 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.409675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.411400 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.411420 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.412800 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.412807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.413056 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.413898 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:42.413924 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:42.417066 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:42.421602 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.421624 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.421634 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.422407 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.422968 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.422973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.422977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.423164 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.424012 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:42.424022 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:42.424266 (+ 244us) client_negotiation.cc:769] Sending connection context
0805 22:47:42.425639 (+ 1373us) client_negotiation.cc:240] Negotiation successful
0805 22:47:42.426069 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":323,"threads_started":1}
I20260805 22:47:42.429031 13048 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.28.61:40937: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473"
member_type: NON_VOTER
last_known_addr {
host: "127.4.28.61"
port: 40937
}
attrs {
promote: true
}
}
dest_uuid: "87fc4f6a5854435f93c862b049b98b9e"
cas_config_opid_index: -1
I20260805 22:47:42.430481 13048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } }
I20260805 22:47:42.433868 13821 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1a8ab594305c439ab3b78b51bbaec473 (127.4.28.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } } }
I20260805 22:47:42.435642 13744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.60.keytab.
Entry for principal oryx/127.4.28.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.60.keytab.
I20260805 22:47:42.437614 13821 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.28.61:40937
I20260805 22:47:42.438627 13687 master.cc:537] Master@127.4.28.61:40937 shutting down...
I20260805 22:47:42.438654 13821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } } }
I20260805 22:47:42.438735 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.60:35143
--webserver_interface=127.4.28.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.60.keytab
--principal=oryx/127.4.28.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45685,127.4.28.61:40937,127.4.28.60:35143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:47:42.438880 13821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:42.448030 13744 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Refusing update from remote peer 87fc4f6a5854435f93c862b049b98b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260805 22:47:42.449271 13820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 22:47:42.453344 13027 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260805 22:47:42.455305 13027 consensus_peers.cc:487] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e -> Peer 1a8ab594305c439ab3b78b51bbaec473 (127.4.28.61:40937): Couldn't send request to peer 1a8ab594305c439ab3b78b51bbaec473. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:47:42.472361 13687 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:42.472883 13687 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:42.473116 13687 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: stopping tablet replica
I20260805 22:47:42.491724 13687 master.cc:552] Master@127.4.28.61:40937 shutdown complete.
I20260805 22:47:42.491899 13687 master_runner.cc:305] Clearing existing system tablet
I20260805 22:47:42.492455 13687 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:42.492578 13687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:42.498374 13687 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260805 22:47:42.500396 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:42.500830 13687 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:42.501139 13687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260805 22:47:42.501267 13687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal
uuid: "1a8ab594305c439ab3b78b51bbaec473"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "79adaebc42401a51bd2d94aa5d678c96"
server_key_iv: "6824317677b71076b77c189a1773170e"
server_key_version: "encryptionkey@0"
I20260805 22:47:42.507186 13687 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:42.515228 13687 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260805 22:47:42.515512 13687 log.cc:1196] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260805 22:47:42.516114 13687 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: Deleting consensus metadata
I20260805 22:47:42.516399 13687 master_runner.cc:315] Copying system tablet from 127.4.28.62:45685
I20260805 22:47:42.519526 13687 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: tablet copy: Beginning tablet copy session from remote peer at address 127.4.28.62:45685
I20260805 22:47:42.537745 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.520702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.4.28.62:45685)
0805 22:47:42.520903 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.520909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.521489 (+ 580us) server_negotiation.cc:407] Connection header received
0805 22:47:42.521720 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.521727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.521832 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.521954 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:42.523063 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.524215 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.526225 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.527217 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.527439 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.530281 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:42.530289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:42.530294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:42.530342 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:42.534191 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.535266 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.535274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.535278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.535476 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:42.535963 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:42.535969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:42.535972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:42.536370 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:42.536571 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:42.537066 (+ 495us) server_negotiation.cc:299] Negotiation successful
0805 22:47:42.537325 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260805 22:47:42.537820 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.520689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53336)
0805 22:47:42.521350 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:42.521371 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:42.521574 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:42.522016 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:42.522022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:42.522594 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:42.522886 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.522901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.524427 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.524433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.526018 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.526038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.527720 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.527727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.527918 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.528696 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:42.528721 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:42.529992 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:42.534422 (+ 4430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.534429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.534434 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.535084 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.535624 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:42.535629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:42.535633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:42.535820 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:42.536521 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:42.536526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:42.536729 (+ 203us) client_negotiation.cc:769] Sending connection context
0805 22:47:42.536907 (+ 178us) client_negotiation.cc:240] Negotiation successful
0805 22:47:42.537286 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":364,"threads_started":1}
I20260805 22:47:42.539129 13073 tablet_copy_service.cc:140] P 87fc4f6a5854435f93c862b049b98b9e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1a8ab594305c439ab3b78b51bbaec473 ({username='oryx', principal='oryx/127.4.28.61@KRBTEST.COM'} at 127.0.0.1:53336)
I20260805 22:47:42.539534 13073 tablet_copy_service.cc:161] P 87fc4f6a5854435f93c862b049b98b9e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1a8ab594305c439ab3b78b51bbaec473 at {username='oryx', principal='oryx/127.4.28.61@KRBTEST.COM'} at 127.0.0.1:53336: session id = 1a8ab594305c439ab3b78b51bbaec473-00000000000000000000000000000000
I20260805 22:47:42.544821 13073 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:47:42.548355 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260805 22:47:42.556517 13687 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: tablet copy: Starting download of 0 data blocks...
I20260805 22:47:42.556999 13687 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: tablet copy: Starting download of 1 WAL segments...
I20260805 22:47:42.561335 13687 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:47:42.566681 13073 tablet_copy_service.cc:342] P 87fc4f6a5854435f93c862b049b98b9e: Request end of tablet copy session 1a8ab594305c439ab3b78b51bbaec473-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.28.61@KRBTEST.COM'} at 127.0.0.1:53336
I20260805 22:47:42.567052 13073 tablet_copy_service.cc:433] P 87fc4f6a5854435f93c862b049b98b9e: ending tablet copy session 1a8ab594305c439ab3b78b51bbaec473-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1a8ab594305c439ab3b78b51bbaec473
I20260805 22:47:42.580744 13687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:42.587322 13833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:42.588158 13834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:42.596481 13836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:42.596771 13687 server_base.cc:1034] running on GCE node
I20260805 22:47:42.597345 13687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:42.598155 13687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:42.599388 13687 hybrid_clock.cc:648] HybridClock initialized: now 1785970062599316 us; error 46 us; skew 500 ppm
I20260805 22:47:42.599644 13687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:42 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970062, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:42.601958 13687 init.cc:377] Logged in from keytab as oryx/127.4.28.61@KRBTEST.COM (short username oryx)
I20260805 22:47:42.608883 13687 webserver.cc:460] Webserver started at http://127.4.28.61:43253/ using document root <none> and password file <none>
I20260805 22:47:42.609303 13687 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:42.609407 13687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:42.612823 13687 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260805 22:47:42.617797 13848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:42.618284 13687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 22:47:42.618423 13687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal
uuid: "1a8ab594305c439ab3b78b51bbaec473"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "79adaebc42401a51bd2d94aa5d678c96"
server_key_iv: "6824317677b71076b77c189a1773170e"
server_key_version: "encryptionkey@0"
I20260805 22:47:42.618602 13687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:42.630856 13687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:42.635267 13687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:42.663757 13687 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.61:40937
I20260805 22:47:42.663794 13921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.61:40937 every 8 connection(s)
I20260805 22:47:42.665149 13687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
I20260805 22:47:42.671279 13924 sys_catalog.cc:263] Verifying existing consensus state
I20260805 22:47:42.675478 13924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: Bootstrap starting.
I20260805 22:47:42.697093 13924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:47:42.697811 13924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: Bootstrap complete.
I20260805 22:47:42.698978 13924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } }
I20260805 22:47:42.699280 13924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1a8ab594305c439ab3b78b51bbaec473, State: Initialized, Role: LEARNER
I20260805 22:47:42.699635 13924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } }
I20260805 22:47:42.700560 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } } }
I20260805 22:47:42.701028 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: This master's current role is: LEARNER
I20260805 22:47:42.701033 13924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:42.711722 13943 catalog_manager.cc:1259] Loaded cluster ID: 709c639440b74ebf857ee564da01e9ad
I20260805 22:47:42.711853 13943 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: loading cluster ID for follower catalog manager: success
I20260805 22:47:42.715720 13943 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: acquiring CA information for follower catalog manager: success
I20260805 22:47:42.718993 13943 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 22:47:42.753410 13679 heartbeater.cc:503] Master 127.4.28.62:45685 was elected leader, sending a full tablet report...
I20260805 22:47:42.856125 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.848966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.4.28.61:40937)
0805 22:47:42.849576 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.849581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.849599 (+ 18us) server_negotiation.cc:407] Connection header received
0805 22:47:42.849664 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.849668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.849744 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.849850 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:42.850369 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.851500 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.854052 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.854981 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.855100 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.855294 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:42.855715 (+ 421us) server_negotiation.cc:299] Negotiation successful
0805 22:47:42.855848 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":321,"threads_started":1}
I20260805 22:47:42.856118 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.848686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.0.0.1:54432)
0805 22:47:42.848933 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:42.848956 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:42.849099 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:42.849936 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:42.849942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:42.849976 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:42.850189 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.850201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.851659 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.851665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.853827 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:42.853847 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.855202 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.855209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:42.855440 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:42.855532 (+ 92us) client_negotiation.cc:769] Sending connection context
0805 22:47:42.855658 (+ 126us) client_negotiation.cc:240] Negotiation successful
0805 22:47:42.855824 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260805 22:47:42.857594 13886 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260805 22:47:42.858312 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87fc4f6a5854435f93c862b049b98b9e. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } } }
I20260805 22:47:42.858610 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: This master's current role is: LEARNER
W20260805 22:47:42.865343 13823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:42.865715 13823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:42.865825 13823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:42.873472 13823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:42.873625 13823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:42.873682 13823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:42.873739 13823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:42.873785 13823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:42.884307 13823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45685,127.4.28.61:40937,127.4.28.60:35143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.60.keytab
--principal=oryx/127.4.28.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.60:35143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.28.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:42.885799 13823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:42.887537 13823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:42.900601 13953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:42.900905 13951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:42.900758 13955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:42.902213 13823 server_base.cc:1034] running on GCE node
I20260805 22:47:42.903295 13823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:42.905318 13823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:42.906632 13823 hybrid_clock.cc:648] HybridClock initialized: now 1785970062906547 us; error 76 us; skew 500 ppm
I20260805 22:47:42.907290 13823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:42 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970062, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:42.911664 13823 init.cc:377] Logged in from keytab as oryx/127.4.28.60@KRBTEST.COM (short username oryx)
I20260805 22:47:42.913686 13823 webserver.cc:460] Webserver started at http://127.4.28.60:40761/ using document root <none> and password file <none>
I20260805 22:47:42.914608 13823 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:42.914759 13823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:42.915098 13823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:42.918565 13823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data/instance:
uuid: "56b590795831486283ab1d37bc2d6120"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "e8076a23587660b966d8f3a15c8cdc30"
server_key_iv: "aa29fc87335ce2f523b9c7d0642bc60f"
server_key_version: "encryptionkey@0"
I20260805 22:47:42.919461 13823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal/instance:
uuid: "56b590795831486283ab1d37bc2d6120"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "e8076a23587660b966d8f3a15c8cdc30"
server_key_iv: "aa29fc87335ce2f523b9c7d0642bc60f"
server_key_version: "encryptionkey@0"
I20260805 22:47:42.925364 13823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260805 22:47:42.929189 13963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:42.930778 13823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:42.930963 13823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal
uuid: "56b590795831486283ab1d37bc2d6120"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "e8076a23587660b966d8f3a15c8cdc30"
server_key_iv: "aa29fc87335ce2f523b9c7d0642bc60f"
server_key_version: "encryptionkey@0"
I20260805 22:47:42.931175 13823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:42.956048 13823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:42.960692 13823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:42.960985 13823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:42.980599 13823 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.60:35143
I20260805 22:47:42.980633 14031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.60:35143 every 8 connection(s)
I20260805 22:47:42.982110 13823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
I20260805 22:47:42.983755 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 13823
I20260805 22:47:42.987238 14035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:42.994551 14035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:47:42 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.60@KRBTEST.COM
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970062, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.60@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:43.009608 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.986361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.4.28.60:35143)
0805 22:47:42.987283 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.987297 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.987355 (+ 58us) server_negotiation.cc:407] Connection header received
0805 22:47:42.989015 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.989039 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.989444 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.989931 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:42.991567 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.993004 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.994420 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:42.995194 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:42.995384 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.000738 (+ 5354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.000754 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.000770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.000828 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.004535 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.005596 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.005604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.005612 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.005823 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.006305 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.006309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.006311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.007094 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.007423 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.007917 (+ 494us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.008355 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":419,"threads_started":1}
I20260805 22:47:43.014220 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.997556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.4.28.62:45685)
0805 22:47:42.997777 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:42.997783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:42.998640 (+ 857us) server_negotiation.cc:407] Connection header received
0805 22:47:42.998860 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:42.998866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:42.998954 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:42.999130 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.000600 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.001722 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.003416 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.004105 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.004250 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.008285 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.008291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.008294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.008331 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.011396 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.012172 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.012178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.012181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.012378 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.012796 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.012801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.012803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.013123 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.013246 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.013765 (+ 519us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.013958 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 22:47:43.014256 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:42.997326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53338)
0805 22:47:42.998448 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:42.998517 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:42.998720 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:42.999191 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:42.999197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.000044 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.000416 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.000431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.001890 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.001894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.003245 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.003262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.004299 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.004304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.004448 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.005102 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.005133 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.008010 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.011560 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.011570 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.011575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.012034 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.012504 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.012509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.012513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.012694 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.013273 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.013280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.013453 (+ 173us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.013673 (+ 220us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.013959 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":631,"threads_started":1}
I20260805 22:47:43.015729 14035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970062, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.60@KRBTEST.COM for oryx/127.4.28.61@KRBTEST.COM
I20260805 22:47:43.028280 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.016537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.0.0.1:54438)
0805 22:47:43.016741 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.016761 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.016861 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.017207 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.017211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.017624 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.017779 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.017788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.018776 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.018781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.019818 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.019831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.020499 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.020503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.020628 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.021227 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.021245 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.023273 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.026286 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.026290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.026293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.026695 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.026999 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.027002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.027004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.027124 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.027568 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.027572 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.027703 (+ 131us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.027807 (+ 104us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.028044 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260805 22:47:43.028337 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.016656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.4.28.61:40937)
0805 22:47:43.016845 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.016851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.016868 (+ 17us) server_negotiation.cc:407] Connection header received
0805 22:47:43.016954 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.016958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.017052 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.017132 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.017898 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.018649 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.019949 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.020357 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.020448 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.023493 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.023498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.023502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.023539 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.026121 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.026788 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.026792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.026795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.026919 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.027201 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.027204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.027206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.027475 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.027585 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.027889 (+ 304us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.028102 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:47:43.029456 14035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970062, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.60@KRBTEST.COM for oryx/127.4.28.60@KRBTEST.COM
I20260805 22:47:43.041931 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.030253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:35143 (local address 127.0.0.1:40032)
0805 22:47:43.030471 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.030487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.030585 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.030826 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.030830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.031162 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.031317 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.031325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.032267 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.032271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.033115 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.033126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.033850 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.033855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.033998 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.034583 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.034617 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.036711 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.039570 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.039576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.039580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.039940 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.040295 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.040300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.040304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.040454 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.040951 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.040956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.041092 (+ 136us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.041210 (+ 118us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.041497 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260805 22:47:43.041970 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.030376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.4.28.60:35143)
0805 22:47:43.030528 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.030533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.030560 (+ 27us) server_negotiation.cc:407] Connection header received
0805 22:47:43.030668 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.030672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.030727 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.030804 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.031432 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.032155 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.033230 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.033696 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.033787 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.036888 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.036893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.036896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.036927 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.039429 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.040051 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.040057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.040060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.040179 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.040564 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.040568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.040570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.040839 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.040971 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.041325 (+ 354us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.041675 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":55}
I20260805 22:47:43.047384 14035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Bootstrap starting.
I20260805 22:47:43.050945 14035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:43.052106 14035 log.cc:824] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:43.055112 14035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: No bootstrap required, opened a new log
I20260805 22:47:43.062357 14035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } }
I20260805 22:47:43.062731 14035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:43.062844 14035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b590795831486283ab1d37bc2d6120, State: Initialized, Role: FOLLOWER
I20260805 22:47:43.063552 14035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } }
I20260805 22:47:43.064746 14042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } } }
I20260805 22:47:43.065141 14042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:47:43.065904 14035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: configured and running, proceeding with master startup.
W20260805 22:47:43.075248 14058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 22:47:43.075464 14058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 22:47:43.079798 13823 master_runner.cc:428] Detected that this master 56b590795831486283ab1d37bc2d6120 is joining an existing cluster
I20260805 22:47:43.079931 13823 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.28.60:35143
I20260805 22:47:43.102273 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.084266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.4.28.61:40937)
0805 22:47:43.084548 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.084555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.085680 (+ 1125us) server_negotiation.cc:407] Connection header received
0805 22:47:43.085937 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.085942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.086022 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.086097 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.087463 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.088612 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.090909 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.091579 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.091685 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.094371 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.094378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.094383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.094434 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.097923 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.098726 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.098731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.098734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.099408 (+ 674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.099937 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.099942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.099946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.100382 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.100533 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.101289 (+ 756us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.101839 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260805 22:47:43.103415 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.084361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.0.0.1:54450)
0805 22:47:43.085555 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.085572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.085780 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.086330 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.086334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.086940 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.087303 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.087314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.088830 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.088835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.090738 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.090750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.091994 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.092001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.092180 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.092927 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.092945 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.094122 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.098090 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.098099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.098103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.098603 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.099599 (+ 996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.099603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.099610 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.099802 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.100616 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.100621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.100916 (+ 295us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.102492 (+ 1576us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.102912 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":88,"thread_start_us":781,"threads_started":1}
I20260805 22:47:43.105016 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.084140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.4.28.62:45685)
0805 22:47:43.084342 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.084349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.085204 (+ 855us) server_negotiation.cc:407] Connection header received
0805 22:47:43.085555 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.085562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.085655 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.085732 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.087445 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.088612 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.091390 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.092078 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.092188 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.095291 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.095298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.095303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.095361 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.099027 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.099792 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.099798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.099801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.099968 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.100530 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.100535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.100538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.100913 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.101045 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.104290 (+ 3245us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.104567 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 22:47:43.106294 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.084157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53352)
0805 22:47:43.085042 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.085073 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.085348 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.085958 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.085963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.086783 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.087239 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.087257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.088816 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.088822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.091177 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.091195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.092530 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.092538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.092714 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.093977 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.093998 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.095047 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.099168 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.099173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.099177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.099640 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.100126 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.100130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.100137 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.100343 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.101143 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.101148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.101437 (+ 289us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.105385 (+ 3948us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.105740 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":506,"thread_start_us":556,"threads_started":1}
I20260805 22:47:43.106812 14067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.086000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:35143 (local address 127.0.0.1:40036)
0805 22:47:43.089993 (+ 3993us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.090022 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.090218 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.090689 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.090694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.091458 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.091819 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.091831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.093336 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.093342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.094972 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.094998 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.096276 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.096283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.096458 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.097055 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.097094 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.098488 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.102611 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.102616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.102620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.103241 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.104385 (+ 1144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.104389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.104393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.104541 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.105387 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.105392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.105650 (+ 258us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.105800 (+ 150us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.106124 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3818,"mutex_wait_us":22,"thread_start_us":3670,"threads_started":1}
I20260805 22:47:43.106909 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.085709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.4.28.60:35143)
0805 22:47:43.085887 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.085893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.090125 (+ 4232us) server_negotiation.cc:407] Connection header received
0805 22:47:43.090339 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.090344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.090407 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.090475 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.091960 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.093135 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.095124 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.095870 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.095965 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.098664 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.098670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.098675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.098708 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.102444 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.103385 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.103391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.103395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.103560 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.104655 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.104659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.104662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.105002 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.105301 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.105974 (+ 673us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.106301 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":56}
I20260805 22:47:43.116112 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 13130
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.62@KRBTEST.COM for oryx/127.4.28.60@KRBTEST.COM
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.1@KRBTEST.COM
I20260805 22:47:43.129900 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.112405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:35143 (local address 127.0.0.1:40044)
0805 22:47:43.112607 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.112628 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.112747 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.113162 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.113168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.113751 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.113979 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.114004 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.115275 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.115282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.116717 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.116738 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.117807 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.117815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.118006 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.118751 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.118777 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.121847 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.126635 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.126642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.126647 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.127239 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.127789 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.127795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.127799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.127970 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.128720 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.128725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.128927 (+ 202us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.129136 (+ 209us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.129497 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260805 22:47:43.129889 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.112519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.4.28.60:35143)
0805 22:47:43.112671 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.112675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.112697 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:47:43.112907 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.112911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.113024 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.113102 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.114166 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.115046 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.116937 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.117609 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.117730 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.122157 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.122164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.122170 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.122217 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.126405 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.127460 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.127466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.127469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.127624 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.128119 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.128125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.128129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.128560 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.128729 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.129220 (+ 491us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.129473 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 22:47:43.131809 13048 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.28.60:35143: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "56b590795831486283ab1d37bc2d6120"
member_type: NON_VOTER
last_known_addr {
host: "127.4.28.60"
port: 35143
}
attrs {
promote: true
}
}
dest_uuid: "87fc4f6a5854435f93c862b049b98b9e"
cas_config_opid_index: 5
I20260805 22:47:43.132803 13048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } }
I20260805 22:47:43.134692 13820 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 56b590795831486283ab1d37bc2d6120 (127.4.28.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.136897 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.117199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.4.28.1:33079)
0805 22:47:43.117883 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.117893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.117924 (+ 31us) server_negotiation.cc:407] Connection header received
0805 22:47:43.118006 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.118015 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.118353 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.118535 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.120079 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.121315 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.122899 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.123624 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.123757 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.128275 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.128290 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.128308 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.128346 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.132444 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.133415 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.133426 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.133430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.133643 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.134162 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.134167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.134170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.134677 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.135065 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.136146 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.136459 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":326,"threads_started":1}
I20260805 22:47:43.138412 13294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:43.139029 13997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:43.141891 13997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 FOLLOWER]: Refusing update from remote peer 87fc4f6a5854435f93c862b049b98b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260805 22:47:43.152756 14069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:43.153268 13820 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.28.60:35143
I20260805 22:47:43.154225 13823 master.cc:537] Master@127.4.28.60:35143 shutting down...
I20260805 22:47:43.155350 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.157871 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:43.157866 13886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 LEARNER]: Refusing update from remote peer 87fc4f6a5854435f93c862b049b98b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260805 22:47:43.159685 14069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:47:43.171751 13886 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 56b590795831486283ab1d37bc2d6120 (127.4.28.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.176522 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: NON_VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: true } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.176837 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: This master's current role is: LEARNER
I20260805 22:47:43.177873 14071 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: attempting to promote NON_VOTER 1a8ab594305c439ab3b78b51bbaec473 to VOTER
I20260805 22:47:43.178902 14071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } }
I20260805 22:47:43.181754 13886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 LEARNER]: Refusing update from remote peer 87fc4f6a5854435f93c862b049b98b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260805 22:47:43.182421 14071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:47:43.184870 14072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1a8ab594305c439ab3b78b51bbaec473 (127.4.28.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.188227 13823 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:43.188689 13823 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:43.188925 13823 tablet_replica.cc:330] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: stopping tablet replica
I20260805 22:47:43.196736 13823 master.cc:552] Master@127.4.28.60:35143 shutdown complete.
I20260805 22:47:43.198042 13823 master_runner.cc:305] Clearing existing system tablet
I20260805 22:47:43.198611 13823 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:43.198902 13823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:43.199286 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.199597 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:43.200946 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.201254 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:43.204483 13886 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1a8ab594305c439ab3b78b51bbaec473 (127.4.28.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.207196 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.206690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:35143 (local address 127.0.0.1:40060)
0805 22:47:43.206884 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.207017 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:35143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260805 22:47:43.207547 13027 consensus_peers.cc:487] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e -> Peer 56b590795831486283ab1d37bc2d6120 (127.4.28.60:35143): Couldn't send request to peer 56b590795831486283ab1d37bc2d6120. Status: Network error: Client connection negotiation failed: client connection to 127.4.28.60:35143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260805 22:47:43.209216 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.209506 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:47:43.209700 13823 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.001s
I20260805 22:47:43.211611 14078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:43.212023 13823 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:43.212482 13823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260805 22:47:43.212643 13823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal
uuid: "56b590795831486283ab1d37bc2d6120"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "e8076a23587660b966d8f3a15c8cdc30"
server_key_iv: "aa29fc87335ce2f523b9c7d0642bc60f"
server_key_version: "encryptionkey@0"
I20260805 22:47:43.219182 13823 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:43.227638 13823 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260805 22:47:43.228009 13823 log.cc:1196] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260805 22:47:43.228659 13823 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Deleting consensus metadata
I20260805 22:47:43.229003 13823 master_runner.cc:315] Copying system tablet from 127.4.28.62:45685
I20260805 22:47:43.231870 13823 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: tablet copy: Beginning tablet copy session from remote peer at address 127.4.28.62:45685
I20260805 22:47:43.250375 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.232937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.4.28.62:45685)
0805 22:47:43.233191 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.233197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.234215 (+ 1018us) server_negotiation.cc:407] Connection header received
0805 22:47:43.234508 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.234515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.234615 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.234686 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.236038 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.237204 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.239341 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.240062 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.240169 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.243188 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.243195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.243201 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.243246 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.246938 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.247862 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.247869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.247873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.248060 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.248542 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.248548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.248551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.248928 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.249088 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.249719 (+ 631us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.249994 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260805 22:47:43.251664 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.232898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53364)
0805 22:47:43.234079 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.234100 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.234350 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.234905 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.234915 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.235514 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.235814 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.235829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.237433 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.237439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.239136 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.239156 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.240467 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.240474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.240677 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.241571 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.241598 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.242899 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.247163 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.247169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.247174 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.247704 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.248215 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.248221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.248225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.248407 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.249178 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.249184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.249374 (+ 190us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.250800 (+ 1426us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.251193 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":775,"threads_started":1}
I20260805 22:47:43.252377 13073 tablet_copy_service.cc:140] P 87fc4f6a5854435f93c862b049b98b9e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 56b590795831486283ab1d37bc2d6120 ({username='oryx', principal='oryx/127.4.28.60@KRBTEST.COM'} at 127.0.0.1:53364)
I20260805 22:47:43.252555 13073 tablet_copy_service.cc:161] P 87fc4f6a5854435f93c862b049b98b9e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 56b590795831486283ab1d37bc2d6120 at {username='oryx', principal='oryx/127.4.28.60@KRBTEST.COM'} at 127.0.0.1:53364: session id = 56b590795831486283ab1d37bc2d6120-00000000000000000000000000000000
I20260805 22:47:43.256415 13073 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:47:43.258581 13823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260805 22:47:43.266147 13823 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: tablet copy: Starting download of 0 data blocks...
I20260805 22:47:43.266526 13823 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: tablet copy: Starting download of 1 WAL segments...
I20260805 22:47:43.269560 13823 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:47:43.274992 13073 tablet_copy_service.cc:342] P 87fc4f6a5854435f93c862b049b98b9e: Request end of tablet copy session 56b590795831486283ab1d37bc2d6120-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.28.60@KRBTEST.COM'} at 127.0.0.1:53364
I20260805 22:47:43.275182 13073 tablet_copy_service.cc:433] P 87fc4f6a5854435f93c862b049b98b9e: ending tablet copy session 56b590795831486283ab1d37bc2d6120-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 56b590795831486283ab1d37bc2d6120
I20260805 22:47:43.288064 13823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:43.291041 14088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:43.294909 14090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:43.302915 14093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:43.303301 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 13130
I20260805 22:47:43.303527 13823 server_base.cc:1034] running on GCE node
I20260805 22:47:43.303891 13823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:43.311153 13823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:43.312523 13823 hybrid_clock.cc:648] HybridClock initialized: now 1785970063312476 us; error 55 us; skew 500 ppm
I20260805 22:47:43.312849 13823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970063, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:43.315320 13823 init.cc:377] Logged in from keytab as oryx/127.4.28.60@KRBTEST.COM (short username oryx)
I20260805 22:47:43.319968 13823 webserver.cc:460] Webserver started at http://127.4.28.60:46653/ using document root <none> and password file <none>
I20260805 22:47:43.320358 13823 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:43.320462 13823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:43.323660 13823 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260805 22:47:43.325788 14102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:43.326284 13823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 22:47:43.326470 13823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal
uuid: "56b590795831486283ab1d37bc2d6120"
format_stamp: "Formatted at 2026-08-05 22:47:42 on dist-test-slave-7lbf"
server_key: "e8076a23587660b966d8f3a15c8cdc30"
server_key_iv: "aa29fc87335ce2f523b9c7d0642bc60f"
server_key_version: "encryptionkey@0"
I20260805 22:47:43.326759 13823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:43.334367 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:33079
--local_ip_for_outbound_sockets=127.4.28.1
--tserver_master_addrs=127.4.28.62:45685
--webserver_port=36929
--webserver_interface=127.4.28.1
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab
--principal=oryx/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.28.62:45685,127.4.28.61:40937
--tserver_master_addrs=127.4.28.62:45685,127.4.28.61:40937,127.4.28.60:35143 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:47:43.354157 13823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:43.358350 13823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:43.391186 13823 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.60:35143
I20260805 22:47:43.391240 14177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.60:35143 every 8 connection(s)
I20260805 22:47:43.392391 13823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
I20260805 22:47:43.398398 14179 sys_catalog.cc:263] Verifying existing consensus state
I20260805 22:47:43.402712 14179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Bootstrap starting.
I20260805 22:47:43.424162 14179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:47:43.424783 14179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Bootstrap complete.
I20260805 22:47:43.426028 14179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } }
I20260805 22:47:43.426355 14179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 56b590795831486283ab1d37bc2d6120, State: Initialized, Role: LEARNER
I20260805 22:47:43.426678 14179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } }
I20260805 22:47:43.427981 14179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:43.427659 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.428448 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: This master's current role is: LEARNER
I20260805 22:47:43.443519 14197 catalog_manager.cc:1259] Loaded cluster ID: 709c639440b74ebf857ee564da01e9ad
I20260805 22:47:43.443647 14197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: loading cluster ID for follower catalog manager: success
I20260805 22:47:43.447618 14197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: acquiring CA information for follower catalog manager: success
I20260805 22:47:43.450327 14197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 22:47:43.714510 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.707745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:35143 (local address 127.0.0.1:40072)
0805 22:47:43.708272 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.708297 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.708463 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.708949 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.708953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.708977 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:43.709150 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.709157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.711060 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.711065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.712273 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.712288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.713906 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.713910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.714056 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.714084 (+ 28us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.714145 (+ 61us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.714257 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":248,"threads_started":1}
I20260805 22:47:43.714699 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.707861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.4.28.60:35143)
0805 22:47:43.708571 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.708579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.708602 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:47:43.708679 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.708685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.708783 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.708917 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:43.709284 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.710859 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.712447 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.713721 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.713870 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.714112 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.714192 (+ 80us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.714346 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":442,"threads_started":1}
I20260805 22:47:43.715729 14148 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260805 22:47:43.716375 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87fc4f6a5854435f93c862b049b98b9e. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:43.716653 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: This master's current role is: LEARNER
W20260805 22:47:43.726138 14107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:43.726436 14107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:43.726570 14107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:43.734212 14107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:43.734349 14107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:43.734520 14107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:43.745064 14107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab
--principal=oryx/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:33079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=36929
--tserver_master_addrs=127.4.28.62:45685,127.4.28.61:40937,127.4.28.60:35143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:43.746660 14107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:43.748136 14107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:43.763396 14208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:43.763346 14213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:43.763418 14210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:43.764343 14107 server_base.cc:1034] running on GCE node
I20260805 22:47:43.765139 14107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:43.766346 14107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:43.767630 14107 hybrid_clock.cc:648] HybridClock initialized: now 1785970063767552 us; error 69 us; skew 500 ppm
I20260805 22:47:43.768000 14107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970063, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:43.771560 14107 init.cc:377] Logged in from keytab as oryx/127.4.28.1@KRBTEST.COM (short username oryx)
I20260805 22:47:43.773200 14107 webserver.cc:460] Webserver started at http://127.4.28.1:36929/ using document root <none> and password file <none>
I20260805 22:47:43.774039 14107 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:43.774197 14107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:43.780499 14107 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260805 22:47:43.783900 14226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:43.785507 14107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:47:43.785763 14107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "2e0b768323044f0b8a357b08ac2054ed"
format_stamp: "Formatted at 2026-08-05 22:47:40 on dist-test-slave-7lbf"
server_key: "5d20f4b16717e718f08ad836b6856c95"
server_key_iv: "521273a1233adcf84b40e994460e562c"
server_key_version: "encryptionkey@0"
I20260805 22:47:43.786890 14107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:43.831077 14107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:43.835991 14107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:43.836290 14107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:43.837414 14107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:43.838959 14107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:43.839077 14107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:43.839198 14107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:43.839277 14107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:43.872614 14107 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:33079
I20260805 22:47:43.872687 14391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:33079 every 8 connection(s)
I20260805 22:47:43.874228 14107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:43.884188 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 14107
I20260805 22:47:43.884593 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 13309
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.2@KRBTEST.COM
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970063, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.61@KRBTEST.COM
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970063, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.60@KRBTEST.COM
I20260805 22:47:43.926209 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.886441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.4.28.2:42539)
0805 22:47:43.901759 (+ 15318us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.901770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.901808 (+ 38us) server_negotiation.cc:407] Connection header received
0805 22:47:43.901892 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.901901 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.902209 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.904765 (+ 2556us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.905047 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.906819 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.908923 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.909915 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.910845 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.916822 (+ 5977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.916842 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.916862 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.916918 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.921227 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.922293 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.922305 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.922309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.922516 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.923032 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.923038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.923041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.923532 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.925233 (+ 1701us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.925438 (+ 205us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.925784 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15097,"thread_start_us":14949,"threads_started":1}
Aug 05 22:47:43 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970063, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:43.928332 13487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:43.934376 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.877385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:56679 (local address 127.4.28.61:40937)
0805 22:47:43.911466 (+ 34081us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.911472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.911522 (+ 50us) server_negotiation.cc:407] Connection header received
0805 22:47:43.911585 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.911591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.911681 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.911803 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.915068 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.916248 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.919483 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.920147 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.920247 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.925804 (+ 5557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.925825 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.925829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.925875 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.929745 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.930910 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.930915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.930919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.931082 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.931837 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.931842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.931846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.932251 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.932397 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.933734 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.933998 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33934,"thread_start_us":37327,"threads_started":1}
I20260805 22:47:43.935171 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.913004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:49599 (local address 127.4.28.60:35143)
0805 22:47:43.913211 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.913217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.914913 (+ 1696us) server_negotiation.cc:407] Connection header received
0805 22:47:43.915156 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.915163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.915246 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.915327 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.916542 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.917627 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.919906 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.920556 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.920666 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.926101 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.926109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.926114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.926163 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.930715 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.931570 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.931576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.931580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.931766 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.932252 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.932257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.932260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.932628 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.932766 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.934598 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.934847 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260805 22:47:43.935706 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.877304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:51109 (local address 127.4.28.62:45685)
0805 22:47:43.893536 (+ 16232us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:43.893544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:43.912644 (+ 19100us) server_negotiation.cc:407] Connection header received
0805 22:47:43.912913 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:43.912919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:43.913009 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:43.913088 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:43.918932 (+ 5844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.920056 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.922104 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.922800 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.922915 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.927751 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:43.927758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:43.927763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:43.927812 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:43.931437 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.932458 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.932465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.932469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.932652 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:43.933170 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:43.933175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:43.933178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:43.933616 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:43.933776 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:43.934833 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0805 22:47:43.935125 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16060,"thread_start_us":16214,"threads_started":1}
I20260805 22:47:43.936635 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.913374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:35143 (local address 127.4.28.1:49599)
0805 22:47:43.914793 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.914819 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.915022 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.915544 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.915548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.916067 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.916362 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.916375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.917816 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.917821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.919703 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.919720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.921037 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.921044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.921237 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.921847 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.921890 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.925826 (+ 3936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.930876 (+ 5050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.930881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.930884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.931416 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.931902 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.931907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.931910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.932106 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.932826 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.932830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.933007 (+ 177us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.933419 (+ 412us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.934007 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"spinlock_wait_cycles":3328,"thread_start_us":1095,"threads_started":1}
I20260805 22:47:43.936712 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.878161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.4.28.1:56679)
0805 22:47:43.892643 (+ 14482us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.892726 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.896022 (+ 3296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.912557 (+ 16535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.912573 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.913587 (+ 1014us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.914830 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.914863 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.916477 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.916482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.919292 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.919307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.920670 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.920681 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.921003 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.921934 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.921957 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.925441 (+ 3484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.929969 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.930011 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.930034 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.930710 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.931231 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.931235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.931239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.931666 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.932480 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.932489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.933007 (+ 518us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.933355 (+ 348us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.935487 (+ 2132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13993,"mutex_wait_us":1473,"spinlock_wait_cycles":1408,"thread_start_us":33813,"threads_started":1}
I20260805 22:47:43.937099 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:43.878161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.1:51109)
0805 22:47:43.912525 (+ 34364us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:43.912555 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:43.912751 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:43.913338 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:43.913343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:43.914007 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:43.918728 (+ 4721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.918742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.920289 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.920293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.921905 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:43.921921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:43.923335 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:43.923345 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:43.923546 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:43.924165 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:43.924189 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:43.927477 (+ 3288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:43.931640 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.931651 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.931655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.932284 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.932791 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:43.932797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:43.932801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:43.933031 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:43.933846 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:43.933852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:43.934033 (+ 181us) client_negotiation.cc:769] Sending connection context
0805 22:47:43.934171 (+ 138us) client_negotiation.cc:240] Negotiation successful
0805 22:47:43.934510 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34161,"mutex_wait_us":4115,"spinlock_wait_cycles":41856,"thread_start_us":15172,"threads_started":1}
I20260805 22:47:43.938745 14393 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45685
I20260805 22:47:43.938813 14392 heartbeater.cc:347] Connected to a master server at 127.4.28.60:35143
I20260805 22:47:43.938856 14394 heartbeater.cc:347] Connected to a master server at 127.4.28.61:40937
I20260805 22:47:43.939146 14393 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:43.939138 14392 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:43.939137 14394 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:43.940446 14392 heartbeater.cc:511] Master 127.4.28.60:35143 requested a full tablet report, sending...
I20260805 22:47:43.940497 14393 heartbeater.cc:511] Master 127.4.28.62:45685 requested a full tablet report, sending...
I20260805 22:47:43.941368 14394 heartbeater.cc:511] Master 127.4.28.61:40937 requested a full tablet report, sending...
I20260805 22:47:43.942917 13048 ts_manager.cc:194] Re-registered known tserver with Master: 2e0b768323044f0b8a357b08ac2054ed (127.4.28.1:33079)
I20260805 22:47:43.943411 13867 ts_manager.cc:194] Registered new tserver with Master: 2e0b768323044f0b8a357b08ac2054ed (127.4.28.1:33079)
I20260805 22:47:43.943434 14136 ts_manager.cc:194] Registered new tserver with Master: 2e0b768323044f0b8a357b08ac2054ed (127.4.28.1:33079)
I20260805 22:47:43.944113 13048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:51109
I20260805 22:47:44.027333 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 13309
I20260805 22:47:44.051918 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:42539
--local_ip_for_outbound_sockets=127.4.28.2
--tserver_master_addrs=127.4.28.62:45685
--webserver_port=32771
--webserver_interface=127.4.28.2
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab
--principal=oryx/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.28.62:45685,127.4.28.61:40937
--tserver_master_addrs=127.4.28.62:45685,127.4.28.61:40937,127.4.28.60:35143 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:47:44.143878 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: NON_VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: true } } }
I20260805 22:47:44.144321 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:44.145926 14413 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: attempting to promote NON_VOTER 56b590795831486283ab1d37bc2d6120 to VOTER
I20260805 22:47:44.146550 14413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } }
I20260805 22:47:44.148538 14148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 LEARNER]: Refusing update from remote peer 87fc4f6a5854435f93c862b049b98b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260805 22:47:44.148823 13886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Refusing update from remote peer 87fc4f6a5854435f93c862b049b98b9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260805 22:47:44.149261 14413 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260805 22:47:44.149698 14414 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260805 22:47:44.152350 14414 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 56b590795831486283ab1d37bc2d6120 (127.4.28.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:44.153074 13886 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 56b590795831486283ab1d37bc2d6120 (127.4.28.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:44.156538 14414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:44.156891 14414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:44.157157 14414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:44.157439 14414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:44.158051 14420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:44.158510 14420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:47:44.161656 14148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 56b590795831486283ab1d37bc2d6120 (127.4.28.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:44.166079 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:44.166395 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:44.166520 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87fc4f6a5854435f93c862b049b98b9e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:44.166821 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: This master's current role is: FOLLOWER
W20260805 22:47:44.417652 14412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:44.417981 14412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:44.418089 14412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:44.425853 14412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:44.425979 14412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:44.426102 14412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:44.436653 14412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab
--principal=oryx/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:42539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=32771
--tserver_master_addrs=127.4.28.62:45685,127.4.28.61:40937,127.4.28.60:35143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:44.438311 14412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:44.439666 14412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:44.453497 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:44.453737 14432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:44.454103 14428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:44.454527 14412 server_base.cc:1034] running on GCE node
I20260805 22:47:44.455230 14412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:44.456265 14412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:44.457502 14412 hybrid_clock.cc:648] HybridClock initialized: now 1785970064457442 us; error 60 us; skew 500 ppm
I20260805 22:47:44.457893 14412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:44 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970064, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:44.461197 14412 init.cc:377] Logged in from keytab as oryx/127.4.28.2@KRBTEST.COM (short username oryx)
I20260805 22:47:44.462884 14412 webserver.cc:460] Webserver started at http://127.4.28.2:32771/ using document root <none> and password file <none>
I20260805 22:47:44.463685 14412 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:44.463836 14412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:44.470089 14412 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:47:44.473101 14442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:44.474614 14412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:44.474792 14412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "2761cac4c2f84655889032c0b97978e1"
format_stamp: "Formatted at 2026-08-05 22:47:41 on dist-test-slave-7lbf"
server_key: "441c4c5a3dd4ae357c29c16be4effda3"
server_key_iv: "6fe6582315278861a0d28b932d12bce7"
server_key_version: "encryptionkey@0"
I20260805 22:47:44.475823 14412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:44.492707 14412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:44.497346 14412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:44.497699 14412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:44.498847 14412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:44.500344 14412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:44.500442 14412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:44.500557 14412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:44.500630 14412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:44.533246 14412 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:42539
I20260805 22:47:44.533296 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:42539 every 8 connection(s)
I20260805 22:47:44.535048 14412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:44.539968 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 14412
I20260805 22:47:44.540316 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 13501
Aug 05 22:47:44 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.3@KRBTEST.COM
Aug 05 22:47:44 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970064, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
Aug 05 22:47:44 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970064, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.60@KRBTEST.COM
Aug 05 22:47:44 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970064, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.61@KRBTEST.COM
I20260805 22:47:44.588223 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:44.538133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:58069 (local address 127.4.28.62:45685)
0805 22:47:44.564292 (+ 26159us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:44.564299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:44.564337 (+ 38us) server_negotiation.cc:407] Connection header received
0805 22:47:44.564409 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:44.564414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:44.564495 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:44.564615 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:44.569609 (+ 4994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.570785 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.573739 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.574435 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.574547 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:44.580101 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:44.580108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:44.580112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:44.580159 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:44.583833 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:44.585046 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:44.585052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:44.585055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:44.585218 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:44.585820 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:44.585826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:44.585829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:44.586199 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:44.586346 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:44.587496 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0805 22:47:44.587748 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25997,"thread_start_us":26600,"threads_started":1}
I20260805 22:47:44.588227 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:44.538132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:39789 (local address 127.4.28.61:40937)
0805 22:47:44.565280 (+ 27148us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:44.565288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:44.566165 (+ 877us) server_negotiation.cc:407] Connection header received
0805 22:47:44.566242 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:44.566248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:44.566358 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:44.566465 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:44.569625 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.570782 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.574671 (+ 3889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.575336 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.575482 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:44.580712 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:44.580720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:44.580725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:44.580788 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:44.584715 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:44.585618 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:44.585624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:44.585627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:44.585792 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:44.586255 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:44.586260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:44.586263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:44.586625 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:44.586779 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:44.587651 (+ 872us) server_negotiation.cc:299] Negotiation successful
0805 22:47:44.587891 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27009,"thread_start_us":26860,"threads_started":1}
I20260805 22:47:44.588541 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:44.558520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:36641 (local address 127.4.28.60:35143)
0805 22:47:44.567872 (+ 9352us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:44.567876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:44.567892 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:47:44.567941 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:44.567944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:44.568001 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:44.568097 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:44.569605 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.570786 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.573744 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.574443 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.574567 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:44.580129 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:44.580135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:44.580140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:44.580179 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:44.583775 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:44.585051 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:44.585057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:44.585060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:44.585215 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:44.585833 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:44.585839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:44.585842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:44.586193 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:44.586345 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:44.587959 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0805 22:47:44.588189 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9244,"thread_start_us":9092,"threads_started":1}
I20260805 22:47:44.589082 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:44.541440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.4.28.3:40595)
0805 22:47:44.566049 (+ 24609us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:44.566058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:44.566091 (+ 33us) server_negotiation.cc:407] Connection header received
0805 22:47:44.566153 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:44.566160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:44.566387 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:44.569238 (+ 2851us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:44.569571 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.571091 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.572441 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.573093 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.573217 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:44.579997 (+ 6780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:44.580019 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:44.580040 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:44.580089 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:44.584656 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:44.585649 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:44.585658 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:44.585661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:44.585814 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:44.586292 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:44.586297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:44.586300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:44.586698 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:44.586989 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:44.588454 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0805 22:47:44.588729 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24435,"thread_start_us":23999,"threads_started":1}
I20260805 22:47:44.590620 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:44.538890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.2:58069)
0805 22:47:44.553509 (+ 14619us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:44.553609 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:44.556476 (+ 2867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:44.568343 (+ 11867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:44.568348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:44.568837 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:44.569359 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:44.569368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.570996 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.571002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:44.573547 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:44.573558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.574930 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.574939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:44.575370 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:44.576683 (+ 1313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:44.576708 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:44.579739 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:44.583992 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:44.584004 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:44.584025 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:44.584876 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:44.585372 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:44.585375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:44.585378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:44.585674 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:44.586436 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:44.586445 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:44.586966 (+ 521us) client_negotiation.cc:769] Sending connection context
0805 22:47:44.588756 (+ 1790us) client_negotiation.cc:240] Negotiation successful
0805 22:47:44.589176 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14154,"mutex_wait_us":980,"spinlock_wait_cycles":10368,"thread_start_us":26460,"threads_started":1}
I20260805 22:47:44.590667 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:44.564497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:35143 (local address 127.4.28.2:36641)
0805 22:47:44.567225 (+ 2728us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:44.567248 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:44.567404 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:44.568165 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:44.568170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:44.568865 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:44.569332 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:44.569368 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.570997 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.571002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:44.573504 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:44.573549 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.574930 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.574939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:44.575370 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:44.576303 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:44.576345 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:44.579881 (+ 3536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:44.583996 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:44.584003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:44.584025 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:44.584881 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:44.585356 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:44.585360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:44.585364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:44.585674 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:44.586436 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:44.586445 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:44.586966 (+ 521us) client_negotiation.cc:769] Sending connection context
0805 22:47:44.587330 (+ 364us) client_negotiation.cc:240] Negotiation successful
0805 22:47:44.589057 (+ 1727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2587,"mutex_wait_us":108,"spinlock_wait_cycles":6528,"thread_start_us":2423,"threads_started":1}
I20260805 22:47:44.590687 13673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:44.591681 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:44.538890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.4.28.2:39789)
0805 22:47:44.565851 (+ 26961us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:44.565872 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:44.566048 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:44.566900 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:44.566916 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:44.568144 (+ 1228us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:44.569340 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:44.569368 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.570999 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.571004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:44.574455 (+ 3451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:44.574472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:44.575618 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:44.575625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:44.575806 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:44.576409 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:44.576429 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:44.580400 (+ 3971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:44.584874 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:44.584879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:44.584883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:44.585456 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:44.585947 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:44.585952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:44.585955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:44.586116 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:44.586846 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:44.586850 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:44.587023 (+ 173us) client_negotiation.cc:769] Sending connection context
0805 22:47:44.587340 (+ 317us) client_negotiation.cc:240] Negotiation successful
0805 22:47:44.589603 (+ 2263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26784,"mutex_wait_us":3254,"spinlock_wait_cycles":34048,"thread_start_us":17962,"threads_started":1}
I20260805 22:47:44.592195 14610 heartbeater.cc:347] Connected to a master server at 127.4.28.60:35143
I20260805 22:47:44.592344 14611 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45685
I20260805 22:47:44.592577 14611 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:44.592576 14610 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:44.593719 14610 heartbeater.cc:511] Master 127.4.28.60:35143 requested a full tablet report, sending...
I20260805 22:47:44.593930 14611 heartbeater.cc:511] Master 127.4.28.62:45685 requested a full tablet report, sending...
I20260805 22:47:44.593935 14612 heartbeater.cc:347] Connected to a master server at 127.4.28.61:40937
I20260805 22:47:44.594113 14612 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:44.594624 14612 heartbeater.cc:511] Master 127.4.28.61:40937 requested a full tablet report, sending...
I20260805 22:47:44.595683 13048 ts_manager.cc:194] Re-registered known tserver with Master: 2761cac4c2f84655889032c0b97978e1 (127.4.28.2:42539)
I20260805 22:47:44.595824 13867 ts_manager.cc:194] Registered new tserver with Master: 2761cac4c2f84655889032c0b97978e1 (127.4.28.2:42539)
I20260805 22:47:44.596839 13048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:58069
I20260805 22:47:44.596968 14136 ts_manager.cc:194] Registered new tserver with Master: 2761cac4c2f84655889032c0b97978e1 (127.4.28.2:42539)
I20260805 22:47:44.700456 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 13501
I20260805 22:47:44.723613 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:40595
--local_ip_for_outbound_sockets=127.4.28.3
--tserver_master_addrs=127.4.28.62:45685
--webserver_port=34723
--webserver_interface=127.4.28.3
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab
--principal=oryx/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.28.62:45685,127.4.28.61:40937
--tserver_master_addrs=127.4.28.62:45685,127.4.28.61:40937,127.4.28.60:35143 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:47:44.947189 14393 heartbeater.cc:503] Master 127.4.28.62:45685 was elected leader, sending a full tablet report...
W20260805 22:47:45.085930 14629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:45.086274 14629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:45.086421 14629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:45.094528 14629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:45.094676 14629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:45.094830 14629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:45.105468 14629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab
--principal=oryx/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:40595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=34723
--tserver_master_addrs=127.4.28.62:45685,127.4.28.61:40937,127.4.28.60:35143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:45.107049 14629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:45.108487 14629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:45.124372 14636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:45.124336 14635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:45.124402 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:45.125205 14629 server_base.cc:1034] running on GCE node
I20260805 22:47:45.125917 14629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:45.126947 14629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:45.128116 14629 hybrid_clock.cc:648] HybridClock initialized: now 1785970065128051 us; error 60 us; skew 500 ppm
I20260805 22:47:45.128475 14629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:45 dist-test-slave-7lbf krb5kdc[12981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970065, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:45.136245 14629 init.cc:377] Logged in from keytab as oryx/127.4.28.3@KRBTEST.COM (short username oryx)
I20260805 22:47:45.137957 14629 webserver.cc:460] Webserver started at http://127.4.28.3:34723/ using document root <none> and password file <none>
I20260805 22:47:45.138780 14629 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:45.138942 14629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:45.145301 14629 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260805 22:47:45.148613 14650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:45.150291 14629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:45.150494 14629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "79ed51ed787046eeba71558825437f70"
format_stamp: "Formatted at 2026-08-05 22:47:41 on dist-test-slave-7lbf"
server_key: "22f330dcf5e36e54448152a2797eee97"
server_key_iv: "8f0166f857b9824cb47ef1afeb0694ad"
server_key_version: "encryptionkey@0"
I20260805 22:47:45.151681 14629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:45.168431 14629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:45.173216 14629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:45.173576 14629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:45.174719 14629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:45.176263 14629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:45.176367 14629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:45.176484 14629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:45.176564 14629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:45.209887 14629 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:40595
I20260805 22:47:45.209962 14811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:40595 every 8 connection(s)
I20260805 22:47:45.211544 14629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:45.212065 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 14629
Aug 05 22:47:45 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970065, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.60@KRBTEST.COM
Aug 05 22:47:45 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970065, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
Aug 05 22:47:45 dist-test-slave-7lbf krb5kdc[12981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970065, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.61@KRBTEST.COM
I20260805 22:47:45.275053 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.235639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:35143 (local address 127.4.28.3:58589)
0805 22:47:45.243444 (+ 7805us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:45.243464 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:45.243584 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:45.245639 (+ 2055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:45.245645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:45.246187 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:45.246425 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.246435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.248607 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.248612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.250346 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.250361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.251550 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.251558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.251955 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.255641 (+ 3686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:45.255668 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:45.260474 (+ 4806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:45.265648 (+ 5174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:45.265661 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:45.265689 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:45.266501 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:45.269109 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:45.269115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:45.269120 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:45.269509 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:45.271257 (+ 1748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:45.271268 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:45.272111 (+ 843us) client_negotiation.cc:769] Sending connection context
0805 22:47:45.272502 (+ 391us) client_negotiation.cc:240] Negotiation successful
0805 22:47:45.273273 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7644,"mutex_wait_us":22,"thread_start_us":8023,"threads_started":1}
I20260805 22:47:45.276561 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.229546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:58589 (local address 127.4.28.60:35143)
0805 22:47:45.245099 (+ 15553us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.245106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.245128 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:47:45.245194 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.245198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.245292 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.245423 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:45.246595 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.248410 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.250584 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.251267 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.251955 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.261088 (+ 9133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:45.261097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:45.261103 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:45.261159 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:45.265318 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.266709 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.266716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.266721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.266949 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.270380 (+ 3431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.270387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.270391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.270893 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:45.271070 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.275795 (+ 4725us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.276138 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15419,"thread_start_us":16962,"threads_started":1}
I20260805 22:47:45.277427 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.215617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:41333 (local address 127.4.28.61:40937)
0805 22:47:45.237358 (+ 21741us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.237365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.238633 (+ 1268us) server_negotiation.cc:407] Connection header received
0805 22:47:45.242298 (+ 3665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.242305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.242419 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.242530 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:45.246031 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.247161 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.250089 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.251223 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.251338 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.264656 (+ 13318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:45.264667 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:45.264672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:45.264729 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:45.268689 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.270526 (+ 1837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.270533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.270537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.270744 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.275053 (+ 4309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.275061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.275064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.275490 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:45.275659 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.276692 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.276992 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21581,"thread_start_us":21752,"threads_started":1}
I20260805 22:47:45.280652 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.215769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:60643 (local address 127.4.28.62:45685)
0805 22:47:45.240287 (+ 24518us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.240293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.240315 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:47:45.242571 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.242577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.242659 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.242779 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:45.245975 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.247161 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.253464 (+ 6303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.254259 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.254402 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.270061 (+ 15659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:45.270069 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:45.270075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:45.270133 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:45.274019 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.278223 (+ 4204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.278231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.278235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.278439 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.278953 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.278959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.278963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.279381 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:45.279540 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.280002 (+ 462us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.280272 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24395,"thread_start_us":24278,"threads_started":1}
I20260805 22:47:45.284940 14813 heartbeater.cc:347] Connected to a master server at 127.4.28.60:35143
I20260805 22:47:45.285380 14813 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:45.286679 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.215896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.4.28.3:41333)
0805 22:47:45.238278 (+ 22382us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:45.238354 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:45.241379 (+ 3025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:45.243163 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:45.243169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:45.244571 (+ 1402us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:45.245769 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.245797 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.247392 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.247397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.249885 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.249902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.251660 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.251666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.252394 (+ 728us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.259491 (+ 7097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:45.259524 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:45.263513 (+ 3989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:45.268988 (+ 5475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:45.268995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:45.269000 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:45.269562 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:45.274243 (+ 4681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:45.274248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:45.274252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:45.274495 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:45.275782 (+ 1287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:45.275788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:45.275968 (+ 180us) client_negotiation.cc:769] Sending connection context
0805 22:47:45.278578 (+ 2610us) client_negotiation.cc:240] Negotiation successful
0805 22:47:45.284644 (+ 6066us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22001,"mutex_wait_us":165,"spinlock_wait_cycles":82944,"thread_start_us":11636,"threads_started":1}
I20260805 22:47:45.286689 14813 heartbeater.cc:511] Master 127.4.28.60:35143 requested a full tablet report, sending...
I20260805 22:47:45.287669 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.215896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.3:60643)
0805 22:47:45.239737 (+ 23841us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:45.239760 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:45.242415 (+ 2655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:45.243112 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:45.243127 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:45.244483 (+ 1356us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:45.245766 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.245795 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.247356 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.247363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.253277 (+ 5914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.253297 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.254456 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.254462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.254651 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.255375 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:45.255421 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:45.269753 (+ 14332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:45.277352 (+ 7599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:45.277359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:45.277364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:45.278034 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:45.278627 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:45.278632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:45.278637 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:45.278816 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:45.279499 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:45.279504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:45.279684 (+ 180us) client_negotiation.cc:769] Sending connection context
0805 22:47:45.279848 (+ 164us) client_negotiation.cc:240] Negotiation successful
0805 22:47:45.287205 (+ 7357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11680,"mutex_wait_us":7118,"spinlock_wait_cycles":7424,"thread_start_us":21630,"threads_started":1}
I20260805 22:47:45.290481 14814 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45685
I20260805 22:47:45.290683 14814 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:45.291538 14814 heartbeater.cc:511] Master 127.4.28.62:45685 requested a full tablet report, sending...
I20260805 22:47:45.292075 14136 ts_manager.cc:194] Registered new tserver with Master: 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595)
I20260805 22:47:45.292842 14817 heartbeater.cc:347] Connected to a master server at 127.4.28.61:40937
I20260805 22:47:45.292865 13048 ts_manager.cc:194] Re-registered known tserver with Master: 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595)
I20260805 22:47:45.293000 14817 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:45.293469 14817 heartbeater.cc:511] Master 127.4.28.61:40937 requested a full tablet report, sending...
I20260805 22:47:45.294241 13048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:60643
I20260805 22:47:45.294837 13867 ts_manager.cc:194] Registered new tserver with Master: 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595)
I20260805 22:47:45.303676 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.249563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.4.28.62:45685)
0805 22:47:45.281201 (+ 31638us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.281208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.281230 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:47:45.281291 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.281297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.281385 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.281480 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:45.285079 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.286218 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.288860 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.289565 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.289687 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.296272 (+ 6585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:45.296279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:45.296284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:45.296331 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:45.300020 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.301061 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.301068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.301072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.301229 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.301754 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.301760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.301763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.302142 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:45.302456 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.303092 (+ 636us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.303339 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31512,"thread_start_us":39812,"threads_started":1}
I20260805 22:47:45.305289 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.252799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.4.28.61:40937)
0805 22:47:45.278056 (+ 25257us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.278062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.280274 (+ 2212us) server_negotiation.cc:407] Connection header received
0805 22:47:45.281921 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.281927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.282017 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.282093 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:45.285469 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.286529 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.289736 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.290409 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.290534 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.298296 (+ 7762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:45.298306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:45.298312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:45.298359 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:45.301820 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.302675 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.302682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.302686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.302860 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.303338 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.303344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.303348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.303751 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:45.304008 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.304658 (+ 650us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.304908 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25133,"thread_start_us":31555,"threads_started":1}
I20260805 22:47:45.307963 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.263454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.4.28.60:35143)
0805 22:47:45.277264 (+ 13810us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.277271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.290471 (+ 13200us) server_negotiation.cc:407] Connection header received
0805 22:47:45.290568 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.290574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.290756 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.290846 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:45.293201 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.294317 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.296949 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.297749 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.297902 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.300819 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:45.300825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:45.300829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:45.300867 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:45.304274 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.305338 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.305344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.305348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.305552 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.306262 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.306268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.306272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.306660 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:45.306977 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.307353 (+ 376us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.307575 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13676,"thread_start_us":21797,"threads_started":1}
I20260805 22:47:45.308869 13048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:45.316397 13048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:45.359915 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.338894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.4.28.3:40595)
0805 22:47:45.339903 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.339911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.347271 (+ 7360us) server_negotiation.cc:407] Connection header received
0805 22:47:45.350050 (+ 2779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.350062 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.350978 (+ 916us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.351952 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:45.352211 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.353303 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.356156 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.358368 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.358466 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.358704 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.359360 (+ 656us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.359566 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":470,"threads_started":1}
I20260805 22:47:45.359925 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.338653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:40595 (local address 127.0.0.1:48438)
0805 22:47:45.346096 (+ 7443us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:45.346118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:45.349731 (+ 3613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:45.351279 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:45.351284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:45.351313 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:45.351718 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.351734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.353456 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.353461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.355954 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.355969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.358909 (+ 2940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.358918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.359142 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.359182 (+ 40us) client_negotiation.cc:769] Sending connection context
0805 22:47:45.359286 (+ 104us) client_negotiation.cc:240] Negotiation successful
0805 22:47:45.359455 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7284,"mutex_wait_us":1,"thread_start_us":7121,"threads_started":1}
I20260805 22:47:45.361754 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.338053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33079 (local address 127.0.0.1:33476)
0805 22:47:45.338790 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:45.338811 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:45.338958 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:45.354689 (+ 15731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:45.354696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:45.354733 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:45.355284 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.355298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.357134 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.357139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.359037 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.359052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.360937 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.360944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.361122 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.361157 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:47:45.361241 (+ 84us) client_negotiation.cc:240] Negotiation successful
0805 22:47:45.361380 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":731,"threads_started":1}
I20260805 22:47:45.363255 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.346296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42539 (local address 127.0.0.1:59684)
0805 22:47:45.355389 (+ 9093us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:45.355410 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:45.355541 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:45.356520 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:45.356525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:45.356553 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:45.356918 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.356930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.358713 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.358719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.360793 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.360809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.362496 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.362503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.362680 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.362711 (+ 31us) client_negotiation.cc:769] Sending connection context
0805 22:47:45.362816 (+ 105us) client_negotiation.cc:240] Negotiation successful
0805 22:47:45.362944 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8934,"thread_start_us":8750,"threads_started":1}
I20260805 22:47:45.363283 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.346612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.4.28.2:42539)
0805 22:47:45.350785 (+ 4173us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.350797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.355675 (+ 4878us) server_negotiation.cc:407] Connection header received
0805 22:47:45.355797 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.355817 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.356087 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.356290 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:45.357111 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.358571 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.361012 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.362268 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.362377 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.362520 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.362852 (+ 332us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.363003 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3931,"thread_start_us":3502,"threads_started":1}
I20260805 22:47:45.364775 14724 tablet_service.cc:1467] Processing CreateTablet for tablet b8739c63795c4180bfb60ff5ec4a77b9 (DEFAULT_TABLE table=test-table [id=e1ecbb4bf4fa49e4a3505a8c677aa168]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:45.366230 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.338636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.4.28.1:33079)
0805 22:47:45.353957 (+ 15321us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.353966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.353998 (+ 32us) server_negotiation.cc:407] Connection header received
0805 22:47:45.354106 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.354114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.354417 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.354663 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:45.355521 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.356940 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.359209 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.360706 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.360828 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.361070 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.361318 (+ 248us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.365912 (+ 4594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15115,"thread_start_us":14982,"threads_started":1}
I20260805 22:47:45.366400 14724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8739c63795c4180bfb60ff5ec4a77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:45.369544 14509 tablet_service.cc:1467] Processing CreateTablet for tablet b8739c63795c4180bfb60ff5ec4a77b9 (DEFAULT_TABLE table=test-table [id=e1ecbb4bf4fa49e4a3505a8c677aa168]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:45.370342 14292 tablet_service.cc:1467] Processing CreateTablet for tablet b8739c63795c4180bfb60ff5ec4a77b9 (DEFAULT_TABLE table=test-table [id=e1ecbb4bf4fa49e4a3505a8c677aa168]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:45.371320 14509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8739c63795c4180bfb60ff5ec4a77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:45.372579 14292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8739c63795c4180bfb60ff5ec4a77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:45.381856 14874 tablet_bootstrap.cc:492] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: Bootstrap starting.
I20260805 22:47:45.385222 14874 tablet_bootstrap.cc:654] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:45.386715 14874 log.cc:824] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:45.388718 14877 tablet_bootstrap.cc:492] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: Bootstrap starting.
I20260805 22:47:45.390789 14874 tablet_bootstrap.cc:492] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: No bootstrap required, opened a new log
I20260805 22:47:45.391434 14874 ts_tablet_manager.cc:1397] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260805 22:47:45.394271 14877 tablet_bootstrap.cc:654] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:45.395859 14877 log.cc:824] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:45.397861 14878 tablet_bootstrap.cc:492] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: Bootstrap starting.
I20260805 22:47:45.403733 14878 tablet_bootstrap.cc:654] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:45.404145 14874 raft_consensus.cc:348] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } }
I20260805 22:47:45.404980 14874 raft_consensus.cc:374] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:45.405185 14874 raft_consensus.cc:729] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ed51ed787046eeba71558825437f70, State: Initialized, Role: FOLLOWER
I20260805 22:47:45.405939 14878 log.cc:824] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:45.406348 14874 consensus_queue.cc:260] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } }
I20260805 22:47:45.409976 14874 ts_tablet_manager.cc:1428] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: Time spent starting tablet: real 0.018s user 0.014s sys 0.001s
I20260805 22:47:45.410135 14814 heartbeater.cc:503] Master 127.4.28.62:45685 was elected leader, sending a full tablet report...
I20260805 22:47:45.411404 14877 tablet_bootstrap.cc:492] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: No bootstrap required, opened a new log
I20260805 22:47:45.411815 14877 ts_tablet_manager.cc:1397] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.000s
I20260805 22:47:45.412555 14878 tablet_bootstrap.cc:492] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: No bootstrap required, opened a new log
I20260805 22:47:45.412918 14878 ts_tablet_manager.cc:1397] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260805 22:47:45.423075 14877 raft_consensus.cc:348] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } }
I20260805 22:47:45.423880 14877 raft_consensus.cc:374] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:45.424002 14877 raft_consensus.cc:729] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2761cac4c2f84655889032c0b97978e1, State: Initialized, Role: FOLLOWER
I20260805 22:47:45.423771 14878 raft_consensus.cc:348] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } }
I20260805 22:47:45.424263 14878 raft_consensus.cc:374] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:45.424436 14878 raft_consensus.cc:729] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e0b768323044f0b8a357b08ac2054ed, State: Initialized, Role: FOLLOWER
I20260805 22:47:45.424773 14877 consensus_queue.cc:260] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } }
I20260805 22:47:45.425087 14878 consensus_queue.cc:260] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } }
I20260805 22:47:45.426144 14611 heartbeater.cc:503] Master 127.4.28.62:45685 was elected leader, sending a full tablet report...
I20260805 22:47:45.426968 14877 ts_tablet_manager.cc:1428] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260805 22:47:45.431788 14878 ts_tablet_manager.cc:1428] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
W20260805 22:47:45.471587 14819 tablet.cc:2369] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:47:45.538100 14615 tablet.cc:2369] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:47:45.628044 14397 tablet.cc:2369] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:47:45.638389 14885 raft_consensus.cc:482] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:45.638734 14885 raft_consensus.cc:504] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } }
I20260805 22:47:45.640632 14885 leader_election.cc:288] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595), 2761cac4c2f84655889032c0b97978e1 (127.4.28.2:42539)
I20260805 22:47:45.649642 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.640997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:56265 (local address 127.4.28.3:40595)
0805 22:47:45.641244 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.641251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.642027 (+ 776us) server_negotiation.cc:407] Connection header received
0805 22:47:45.642376 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.642383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.642455 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.642522 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:45.643298 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.644291 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.647132 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.648423 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.648582 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.648707 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.649185 (+ 478us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.649340 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:47:45.649643 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.642176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:46507 (local address 127.4.28.2:42539)
0805 22:47:45.642372 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.642378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.643025 (+ 647us) server_negotiation.cc:407] Connection header received
0805 22:47:45.643225 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.643231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.643313 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.643408 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:45.643912 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.644808 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.647280 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.648523 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.648656 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.648742 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.649181 (+ 439us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.649342 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260805 22:47:45.649801 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.642012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42539 (local address 127.4.28.1:46507)
0805 22:47:45.642901 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:45.642917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:45.643073 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:45.643424 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:45.643428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:45.643451 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:45.643739 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.643748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.644974 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.644981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.647103 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.647117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.648739 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.648745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.648929 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.648978 (+ 49us) client_negotiation.cc:769] Sending connection context
0805 22:47:45.649091 (+ 113us) client_negotiation.cc:240] Negotiation successful
0805 22:47:45.649267 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":10,"thread_start_us":588,"threads_started":1}
I20260805 22:47:45.649827 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.641043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:40595 (local address 127.4.28.1:56265)
0805 22:47:45.641919 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:45.641939 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:45.642215 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:45.642726 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:45.642731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:45.642760 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:45.643147 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.643157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.644420 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.644426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.646916 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:45.646937 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.648712 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.648723 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:45.649014 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.649040 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 22:47:45.649115 (+ 75us) client_negotiation.cc:240] Negotiation successful
0805 22:47:45.649269 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":27,"thread_start_us":501,"threads_started":1}
I20260805 22:47:45.651120 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8739c63795c4180bfb60ff5ec4a77b9" candidate_uuid: "2e0b768323044f0b8a357b08ac2054ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ed51ed787046eeba71558825437f70" is_pre_election: true
I20260805 22:47:45.651117 14535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8739c63795c4180bfb60ff5ec4a77b9" candidate_uuid: "2e0b768323044f0b8a357b08ac2054ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2761cac4c2f84655889032c0b97978e1" is_pre_election: true
I20260805 22:47:45.651649 14750 raft_consensus.cc:2454] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e0b768323044f0b8a357b08ac2054ed in term 0.
I20260805 22:47:45.651669 14535 raft_consensus.cc:2454] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e0b768323044f0b8a357b08ac2054ed in term 0.
I20260805 22:47:45.652346 14228 leader_election.cc:302] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e0b768323044f0b8a357b08ac2054ed, 79ed51ed787046eeba71558825437f70; no voters:
I20260805 22:47:45.652781 14885 raft_consensus.cc:2781] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:45.652926 14885 raft_consensus.cc:482] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:45.653043 14885 raft_consensus.cc:3037] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:45.655573 14885 raft_consensus.cc:504] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } }
I20260805 22:47:45.656237 14885 leader_election.cc:288] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [CANDIDATE]: Term 1 election: Requested vote from peers 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595), 2761cac4c2f84655889032c0b97978e1 (127.4.28.2:42539)
I20260805 22:47:45.656802 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8739c63795c4180bfb60ff5ec4a77b9" candidate_uuid: "2e0b768323044f0b8a357b08ac2054ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ed51ed787046eeba71558825437f70"
I20260805 22:47:45.656991 14535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8739c63795c4180bfb60ff5ec4a77b9" candidate_uuid: "2e0b768323044f0b8a357b08ac2054ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2761cac4c2f84655889032c0b97978e1"
I20260805 22:47:45.657065 14750 raft_consensus.cc:3037] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:45.657231 14535 raft_consensus.cc:3037] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:45.660516 14750 raft_consensus.cc:2454] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e0b768323044f0b8a357b08ac2054ed in term 1.
I20260805 22:47:45.660521 14535 raft_consensus.cc:2454] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e0b768323044f0b8a357b08ac2054ed in term 1.
I20260805 22:47:45.661072 14228 leader_election.cc:302] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2761cac4c2f84655889032c0b97978e1, 2e0b768323044f0b8a357b08ac2054ed; no voters:
I20260805 22:47:45.661404 14885 raft_consensus.cc:2781] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:45.661964 14885 raft_consensus.cc:686] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 1 LEADER]: Becoming Leader. State: Replica: 2e0b768323044f0b8a357b08ac2054ed, State: Running, Role: LEADER
I20260805 22:47:45.662585 14885 consensus_queue.cc:237] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } }
I20260805 22:47:45.668828 13048 catalog_manager.cc:5495] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e0b768323044f0b8a357b08ac2054ed (127.4.28.1). New cstate: current_term: 1 leader_uuid: "2e0b768323044f0b8a357b08ac2054ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } health_report { overall_health: UNKNOWN } } }
I20260805 22:47:45.723079 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.717382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.4.28.1:33079)
0805 22:47:45.717642 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.717649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.717670 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:47:45.717783 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.717787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.717875 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.718021 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:45.718458 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.719293 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.720730 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.721150 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.721264 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.721848 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:45.722416 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:45.722612 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.722704 (+ 92us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.722798 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260805 22:47:45.732954 14750 raft_consensus.cc:1261] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [term 1 FOLLOWER]: Refusing update from remote peer 2e0b768323044f0b8a357b08ac2054ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:45.732913 14535 raft_consensus.cc:1261] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [term 1 FOLLOWER]: Refusing update from remote peer 2e0b768323044f0b8a357b08ac2054ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:45.734246 14885 consensus_queue.cc:1035] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:45.734637 14889 consensus_queue.cc:1035] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:45.750236 14892 mvcc.cc:205] Tried to move back new op lower bound from 7315333389229268992 to 7315333388957945856. Current Snapshot: MvccSnapshot[applied={T|T < 7315333389229268992}]
I20260805 22:47:45.750670 14894 mvcc.cc:205] Tried to move back new op lower bound from 7315333389229268992 to 7315333388957945856. Current Snapshot: MvccSnapshot[applied={T|T < 7315333389229268992}]
I20260805 22:47:45.750897 14893 mvcc.cc:205] Tried to move back new op lower bound from 7315333389229268992 to 7315333388957945856. Current Snapshot: MvccSnapshot[applied={T|T < 7315333389229268992}]
I20260805 22:47:45.773445 13046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:45.773906 13046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:45.783126 13046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87fc4f6a5854435f93c862b049b98b9e: Sending DeleteTablet for 3 replicas of tablet b8739c63795c4180bfb60ff5ec4a77b9
I20260805 22:47:45.784406 14292 tablet_service.cc:1514] Processing DeleteTablet for tablet b8739c63795c4180bfb60ff5ec4a77b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:45 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:33476
I20260805 22:47:45.784626 14724 tablet_service.cc:1514] Processing DeleteTablet for tablet b8739c63795c4180bfb60ff5ec4a77b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:45 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:48438
I20260805 22:47:45.784667 14509 tablet_service.cc:1514] Processing DeleteTablet for tablet b8739c63795c4180bfb60ff5ec4a77b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:45 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:59684
I20260805 22:47:45.796020 14907 tablet_replica.cc:330] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: stopping tablet replica
I20260805 22:47:45.796656 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 13000
I20260805 22:47:45.796782 14907 raft_consensus.cc:2229] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:45.797500 14907 raft_consensus.cc:2258] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:45.798138 13086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:45.798440 14904 tablet_replica.cc:330] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: stopping tablet replica
I20260805 22:47:45.799050 14904 raft_consensus.cc:2229] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:45.799928 14904 raft_consensus.cc:2258] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:45.802938 14904 ts_tablet_manager.cc:1905] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:45.804538 14906 tablet_replica.cc:330] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: stopping tablet replica
I20260805 22:47:45.808089 14907 ts_tablet_manager.cc:1905] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:45.808563 14906 raft_consensus.cc:2229] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:45.809428 14906 raft_consensus.cc:2258] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:45.813198 14904 ts_tablet_manager.cc:1918] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:45.813537 14904 log.cc:1196] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/b8739c63795c4180bfb60ff5ec4a77b9
I20260805 22:47:45.814355 14904 ts_tablet_manager.cc:1939] T b8739c63795c4180bfb60ff5ec4a77b9 P 2e0b768323044f0b8a357b08ac2054ed: Deleting consensus metadata
I20260805 22:47:45.816514 13027 catalog_manager.cc:4841] TS 2e0b768323044f0b8a357b08ac2054ed (127.4.28.1:33079): tablet b8739c63795c4180bfb60ff5ec4a77b9 (table test-table [id=e1ecbb4bf4fa49e4a3505a8c677aa168]) successfully deleted
I20260805 22:47:45.816869 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.788069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.4.28.61:40937)
0805 22:47:45.788282 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.788289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.796161 (+ 7872us) server_negotiation.cc:407] Connection header received
0805 22:47:45.796259 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.796265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.796368 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.796482 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:45.799173 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.800410 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.804527 (+ 4117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.805042 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.805185 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.808784 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:45.808790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:45.808794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:45.808844 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:45.812522 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.813631 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.813639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.813643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.813844 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.814418 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.814423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.814427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.814846 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:45.815106 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.816222 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.816500 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:47:45.818018 14906 ts_tablet_manager.cc:1905] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:45.820568 14907 ts_tablet_manager.cc:1918] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:45.820904 14907 log.cc:1196] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/b8739c63795c4180bfb60ff5ec4a77b9
I20260805 22:47:45.821648 14907 ts_tablet_manager.cc:1939] T b8739c63795c4180bfb60ff5ec4a77b9 P 2761cac4c2f84655889032c0b97978e1: Deleting consensus metadata
I20260805 22:47:45.827797 14906 ts_tablet_manager.cc:1918] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:45.828015 14906 log.cc:1196] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/b8739c63795c4180bfb60ff5ec4a77b9
I20260805 22:47:45.828771 14906 ts_tablet_manager.cc:1939] T b8739c63795c4180bfb60ff5ec4a77b9 P 79ed51ed787046eeba71558825437f70: Deleting consensus metadata
I20260805 22:47:45.836891 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:45.795871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.4.28.60:35143)
0805 22:47:45.796091 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:45.796098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:45.818585 (+ 22487us) server_negotiation.cc:407] Connection header received
0805 22:47:45.818856 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:45.818862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:45.818971 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:45.819047 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:45.821366 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.822535 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.824609 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:45.825229 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:45.825376 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:45.829160 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:45.829169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:45.829174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:45.829228 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:45.833099 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.834147 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.834153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.834157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.834349 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:45.834848 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:45.834853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:45.834857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:45.835388 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:45.835662 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:45.836056 (+ 394us) server_negotiation.cc:299] Negotiation successful
0805 22:47:45.836311 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260805 22:47:45.967819 13022 catalog_manager.cc:4841] TS 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595): tablet b8739c63795c4180bfb60ff5ec4a77b9 (table test-table [id=e1ecbb4bf4fa49e4a3505a8c677aa168]) successfully deleted
I20260805 22:47:45.968223 13022 catalog_manager.cc:4841] TS 2761cac4c2f84655889032c0b97978e1 (127.4.28.2:42539): tablet b8739c63795c4180bfb60ff5ec4a77b9 (table test-table [id=e1ecbb4bf4fa49e4a3505a8c677aa168]) successfully deleted
I20260805 22:47:45.968868 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 13000
I20260805 22:47:46.026022 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:46.007824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.4.28.61:40937)
0805 22:47:46.008047 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:46.008054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:46.009099 (+ 1045us) server_negotiation.cc:407] Connection header received
0805 22:47:46.009313 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:46.009319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:46.009413 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:46.009544 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:46.011468 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:46.012259 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:46.014185 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:46.014671 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:46.014792 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:46.018500 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:46.018506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:46.018509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:46.018551 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:46.022550 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:46.023587 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:46.023594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:46.023598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:46.023775 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:46.024247 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:46.024252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:46.024255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:46.024691 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:46.024940 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:46.025379 (+ 439us) server_negotiation.cc:299] Negotiation successful
0805 22:47:46.025670 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260805 22:47:46.026743 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:46.009262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.4.28.60:35143)
0805 22:47:46.009551 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:46.009557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:46.009871 (+ 314us) server_negotiation.cc:407] Connection header received
0805 22:47:46.010051 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:46.010057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:46.010142 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:46.010250 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:46.012427 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:46.013684 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:46.015526 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:46.015992 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:46.016097 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:46.019575 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:46.019583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:46.019588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:46.019637 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:46.023097 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:46.023995 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:46.023999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:46.024002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:46.024139 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:46.024583 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:46.024596 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:46.024599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:46.025022 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:46.025297 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:46.025708 (+ 411us) server_negotiation.cc:299] Negotiation successful
0805 22:47:46.026262 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260805 22:47:46.758239 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:46.757256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.1:57467)
0805 22:47:46.757834 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:46.758009 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":313,"threads_started":1}
W20260805 22:47:46.758719 14393 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:45685 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
I20260805 22:47:46.760090 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:46.759300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.2:53303)
0805 22:47:46.759777 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:46.759928 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":246,"threads_started":1}
W20260805 22:47:46.760465 14611 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:45685 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
I20260805 22:47:46.765900 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:46.765013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.3:37013)
0805 22:47:46.765531 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:46.765679 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":234,"threads_started":1}
W20260805 22:47:46.766283 14814 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:45685 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
I20260805 22:47:47.282101 14922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 87fc4f6a5854435f93c862b049b98b9e)
I20260805 22:47:47.282413 14922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } }
I20260805 22:47:47.283083 14923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 87fc4f6a5854435f93c862b049b98b9e)
I20260805 22:47:47.283337 14923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } }
I20260805 22:47:47.284085 14922 leader_election.cc:288] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87fc4f6a5854435f93c862b049b98b9e (127.4.28.62:45685), 1a8ab594305c439ab3b78b51bbaec473 (127.4.28.61:40937)
I20260805 22:47:47.284745 14923 leader_election.cc:288] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87fc4f6a5854435f93c862b049b98b9e (127.4.28.62:45685), 56b590795831486283ab1d37bc2d6120 (127.4.28.60:35143)
I20260805 22:47:47.285367 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.284278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53834)
0805 22:47:47.284864 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.285067 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":431,"threads_started":1}
W20260805 22:47:47.285894 14105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
I20260805 22:47:47.287930 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.287324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53848)
0805 22:47:47.287510 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.287694 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260805 22:47:47.288187 14105 leader_election.cc:334] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 87fc4f6a5854435f93c862b049b98b9e (127.4.28.62:45685): Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
I20260805 22:47:47.289299 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.287004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53846)
0805 22:47:47.288766 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.288955 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1649,"thread_start_us":1519,"threads_started":1}
W20260805 22:47:47.289659 13851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
I20260805 22:47:47.291530 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.291050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53860)
0805 22:47:47.291223 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.291368 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260805 22:47:47.291810 13851 leader_election.cc:334] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 87fc4f6a5854435f93c862b049b98b9e (127.4.28.62:45685): Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
I20260805 22:47:47.293295 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.284537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.4.28.61:40937)
0805 22:47:47.285350 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.285357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.285991 (+ 634us) server_negotiation.cc:407] Connection header received
0805 22:47:47.286308 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.286314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.286396 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.286473 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:47.287273 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.288520 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.291127 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.292273 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.292379 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.292476 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.292847 (+ 371us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.293006 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":686,"thread_start_us":533,"threads_started":1}
I20260805 22:47:47.293304 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.284341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40937 (local address 127.0.0.1:54494)
0805 22:47:47.285854 (+ 1513us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.285878 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:47.286131 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:47.286736 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:47.286743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:47.286772 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:47.287087 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.287099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.288746 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.288753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.290916 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.290933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.292463 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.292470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.292666 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.292701 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:47:47.292802 (+ 101us) client_negotiation.cc:240] Negotiation successful
0805 22:47:47.292949 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"mutex_wait_us":2,"thread_start_us":2927,"threads_started":1}
I20260805 22:47:47.294380 13886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56b590795831486283ab1d37bc2d6120" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1a8ab594305c439ab3b78b51bbaec473" is_pre_election: true
I20260805 22:47:47.294754 13886 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56b590795831486283ab1d37bc2d6120 in term 1.
I20260805 22:47:47.295451 14931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.287573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.4.28.60:35143)
0805 22:47:47.288548 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.288554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.288577 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:47:47.288636 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.288640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.288716 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.288816 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:47.289509 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.290782 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.293422 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.294456 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.294575 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.294675 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.295004 (+ 329us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.295136 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":860,"thread_start_us":722,"threads_started":1}
I20260805 22:47:47.295488 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.285400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:35143 (local address 127.0.0.1:40102)
0805 22:47:47.286020 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.286039 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:47.286171 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:47.288860 (+ 2689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:47.288865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:47.288907 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:47.289290 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.289300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.290918 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.290924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.293217 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.293234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.294658 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.294664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.294851 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.294884 (+ 33us) client_negotiation.cc:769] Sending connection context
0805 22:47:47.294984 (+ 100us) client_negotiation.cc:240] Negotiation successful
0805 22:47:47.295114 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":43,"thread_start_us":871,"threads_started":1}
I20260805 22:47:47.295360 14106 leader_election.cc:302] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a8ab594305c439ab3b78b51bbaec473, 56b590795831486283ab1d37bc2d6120; no voters: 87fc4f6a5854435f93c862b049b98b9e
I20260805 22:47:47.295801 14922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:47:47.295954 14922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 87fc4f6a5854435f93c862b049b98b9e)
I20260805 22:47:47.296072 14922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:47:47.296460 14148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a8ab594305c439ab3b78b51bbaec473" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56b590795831486283ab1d37bc2d6120" is_pre_election: true
I20260805 22:47:47.298441 14922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } }
I20260805 22:47:47.299018 14148 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1a8ab594305c439ab3b78b51bbaec473 in current term 2: Already voted for candidate 56b590795831486283ab1d37bc2d6120 in this term.
I20260805 22:47:47.299113 14922 leader_election.cc:288] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [CANDIDATE]: Term 2 election: Requested vote from peers 87fc4f6a5854435f93c862b049b98b9e (127.4.28.62:45685), 1a8ab594305c439ab3b78b51bbaec473 (127.4.28.61:40937)
I20260805 22:47:47.299738 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.299315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53862)
0805 22:47:47.299484 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.299577 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260805 22:47:47.299778 13886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56b590795831486283ab1d37bc2d6120" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1a8ab594305c439ab3b78b51bbaec473"
I20260805 22:47:47.300046 13886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:47:47.300372 13852 leader_election.cc:302] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a8ab594305c439ab3b78b51bbaec473; no voters: 56b590795831486283ab1d37bc2d6120, 87fc4f6a5854435f93c862b049b98b9e
I20260805 22:47:47.300820 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.300413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53874)
0805 22:47:47.300590 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.300679 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260805 22:47:47.301004 14105 leader_election.cc:334] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 87fc4f6a5854435f93c862b049b98b9e (127.4.28.62:45685): Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
I20260805 22:47:47.302594 13886 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56b590795831486283ab1d37bc2d6120 in term 2.
I20260805 22:47:47.302769 14923 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:47:47.303074 14106 leader_election.cc:302] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a8ab594305c439ab3b78b51bbaec473, 56b590795831486283ab1d37bc2d6120; no voters: 87fc4f6a5854435f93c862b049b98b9e
I20260805 22:47:47.303417 14922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:47:47.303965 14922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [term 2 LEADER]: Becoming Leader. State: Replica: 56b590795831486283ab1d37bc2d6120, State: Running, Role: LEADER
I20260805 22:47:47.304446 14922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } }
I20260805 22:47:47.306723 14933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56b590795831486283ab1d37bc2d6120. Latest consensus state: current_term: 2 leader_uuid: "56b590795831486283ab1d37bc2d6120" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:47.307123 14933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:47.307776 14935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:47.312033 14935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:47.312961 14935 catalog_manager.cc:1259] Loaded cluster ID: 709c639440b74ebf857ee564da01e9ad
I20260805 22:47:47.313045 14935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:47.314373 14935 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:47.315215 14935 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Loaded TSK: 0
I20260805 22:47:47.316218 14935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:47:47.324860 14132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:47.332472 14132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:47.337311 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.336852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53896)
0805 22:47:47.337047 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.337148 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260805 22:47:47.337419 13886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [term 2 FOLLOWER]: Refusing update from remote peer 56b590795831486283ab1d37bc2d6120: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260805 22:47:47.338287 14922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260805 22:47:47.338460 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.338052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:53902)
0805 22:47:47.338202 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.338277 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260805 22:47:47.338631 14105 consensus_peers.cc:487] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 -> Peer 87fc4f6a5854435f93c862b049b98b9e (127.4.28.62:45685): Couldn't send request to peer 87fc4f6a5854435f93c862b049b98b9e. Status: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260805 22:47:47.341223 14923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56b590795831486283ab1d37bc2d6120. Latest consensus state: current_term: 2 leader_uuid: "56b590795831486283ab1d37bc2d6120" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:47.341540 14923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:47:47.343112 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "56b590795831486283ab1d37bc2d6120" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:47.343577 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:47.343698 14923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "56b590795831486283ab1d37bc2d6120" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "87fc4f6a5854435f93c862b049b98b9e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45685 } } peers { permanent_uuid: "1a8ab594305c439ab3b78b51bbaec473" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40937 } attrs { promote: false } } peers { permanent_uuid: "56b590795831486283ab1d37bc2d6120" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 35143 } attrs { promote: false } } }
I20260805 22:47:47.343978 14923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8ab594305c439ab3b78b51bbaec473 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:47:47.374271 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.353461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.4.28.1:33079)
0805 22:47:47.365822 (+ 12361us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.365829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.366557 (+ 728us) server_negotiation.cc:407] Connection header received
0805 22:47:47.366922 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.366931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.367023 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.367101 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:47.368037 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.369118 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.371707 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.372899 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.373001 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.373106 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.373727 (+ 621us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.373906 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12197,"thread_start_us":12372,"threads_started":1}
I20260805 22:47:47.374850 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.352461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:40595 (local address 127.0.0.1:48444)
0805 22:47:47.352666 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.352686 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:47.352802 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:47.368859 (+ 16057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:47.368865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:47.368895 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:47.369073 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.369087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.370612 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.370619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.372527 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.372542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.373931 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.373938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.374252 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.374287 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:47:47.374384 (+ 97us) client_negotiation.cc:240] Negotiation successful
0805 22:47:47.374526 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":1}
I20260805 22:47:47.375562 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.352601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.4.28.3:40595)
0805 22:47:47.368567 (+ 15966us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.368572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.368588 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:47:47.368647 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.368651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.368711 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.368815 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:47.369544 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.370435 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.372745 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.373754 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.374018 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.374120 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.374874 (+ 754us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.375021 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15831,"thread_start_us":15714,"threads_started":1}
I20260805 22:47:47.375574 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.353270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33079 (local address 127.0.0.1:33506)
0805 22:47:47.366423 (+ 13153us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.366448 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:47.366713 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:47.367348 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:47.367355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:47.367385 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:47.367812 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.367826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.369349 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.369354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.371494 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.371510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.373311 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.373318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.373492 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.373551 (+ 59us) client_negotiation.cc:769] Sending connection context
0805 22:47:47.373650 (+ 99us) client_negotiation.cc:240] Negotiation successful
0805 22:47:47.373800 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13009,"thread_start_us":12864,"threads_started":1}
I20260805 22:47:47.376031 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.352876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42539 (local address 127.0.0.1:59688)
0805 22:47:47.353047 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.353070 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:47.353181 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:47.370070 (+ 16889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:47.370076 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:47.370106 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:47.370459 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.370470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.371504 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.371510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.372971 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.372986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.375023 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.375029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.375273 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.375307 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 22:47:47.375381 (+ 74us) client_negotiation.cc:240] Negotiation successful
0805 22:47:47.375529 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260805 22:47:47.376394 14724 tablet_service.cc:1467] Processing CreateTablet for tablet ca75dae18db14c45bc7fb9967679985f (DEFAULT_TABLE table=test-table [id=3883579c827348759fb8c524b9e3243f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:47.376605 14292 tablet_service.cc:1467] Processing CreateTablet for tablet ca75dae18db14c45bc7fb9967679985f (DEFAULT_TABLE table=test-table [id=3883579c827348759fb8c524b9e3243f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:47.376885 14724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca75dae18db14c45bc7fb9967679985f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:47.377054 14292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca75dae18db14c45bc7fb9967679985f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:47.378391 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.353224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.4.28.2:42539)
0805 22:47:47.369720 (+ 16496us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.369726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.369747 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:47:47.369817 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.369822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.369901 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.370012 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:47.370652 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.371366 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.373182 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.374770 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.377758 (+ 2988us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.377882 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.377943 (+ 61us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.378095 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16240,"thread_start_us":16117,"threads_started":1}
I20260805 22:47:47.382392 14950 tablet_bootstrap.cc:492] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: Bootstrap starting.
I20260805 22:47:47.383323 14509 tablet_service.cc:1467] Processing CreateTablet for tablet ca75dae18db14c45bc7fb9967679985f (DEFAULT_TABLE table=test-table [id=3883579c827348759fb8c524b9e3243f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:47.383563 14951 tablet_bootstrap.cc:492] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: Bootstrap starting.
I20260805 22:47:47.383823 14509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca75dae18db14c45bc7fb9967679985f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:47.385353 14951 tablet_bootstrap.cc:654] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:47.387128 14950 tablet_bootstrap.cc:654] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:47.387868 14951 tablet_bootstrap.cc:492] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: No bootstrap required, opened a new log
I20260805 22:47:47.388139 14951 ts_tablet_manager.cc:1397] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260805 22:47:47.389003 14950 tablet_bootstrap.cc:492] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: No bootstrap required, opened a new log
I20260805 22:47:47.389150 14950 ts_tablet_manager.cc:1397] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260805 22:47:47.389061 14951 raft_consensus.cc:348] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } }
I20260805 22:47:47.389343 14951 raft_consensus.cc:374] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:47.389449 14951 raft_consensus.cc:729] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e0b768323044f0b8a357b08ac2054ed, State: Initialized, Role: FOLLOWER
I20260805 22:47:47.389725 14951 consensus_queue.cc:260] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } }
I20260805 22:47:47.389966 14950 raft_consensus.cc:348] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } }
I20260805 22:47:47.390235 14950 raft_consensus.cc:374] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:47.390364 14950 raft_consensus.cc:729] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ed51ed787046eeba71558825437f70, State: Initialized, Role: FOLLOWER
I20260805 22:47:47.390744 14950 consensus_queue.cc:260] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } }
I20260805 22:47:47.397825 14951 ts_tablet_manager.cc:1428] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260805 22:47:47.398463 14957 tablet_bootstrap.cc:492] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: Bootstrap starting.
I20260805 22:47:47.400962 14957 tablet_bootstrap.cc:654] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:47.402109 14392 heartbeater.cc:503] Master 127.4.28.60:35143 was elected leader, sending a full tablet report...
I20260805 22:47:47.408593 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.391930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.1:40965)
0805 22:47:47.408108 (+ 16178us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.408273 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16012,"threads_started":1}
I20260805 22:47:47.409826 14950 ts_tablet_manager.cc:1428] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: Time spent starting tablet: real 0.021s user 0.011s sys 0.009s
I20260805 22:47:47.410097 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.409663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.1:35051)
0805 22:47:47.409869 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.409957 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260805 22:47:47.414116 14813 heartbeater.cc:503] Master 127.4.28.60:35143 was elected leader, sending a full tablet report...
I20260805 22:47:47.414508 14957 tablet_bootstrap.cc:492] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: No bootstrap required, opened a new log
I20260805 22:47:47.414669 14957 ts_tablet_manager.cc:1397] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260805 22:47:47.415729 14957 raft_consensus.cc:348] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } }
I20260805 22:47:47.416082 14957 raft_consensus.cc:374] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:47.416229 14957 raft_consensus.cc:729] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2761cac4c2f84655889032c0b97978e1, State: Initialized, Role: FOLLOWER
I20260805 22:47:47.416584 14957 consensus_queue.cc:260] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } }
I20260805 22:47:47.418529 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.410416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.3:57741)
0805 22:47:47.418091 (+ 7675us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.418229 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7509,"thread_start_us":7773,"threads_started":1}
I20260805 22:47:47.419603 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.419242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.3:51695)
0805 22:47:47.419421 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.419492 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260805 22:47:47.422940 14957 ts_tablet_manager.cc:1428] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260805 22:47:47.423486 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.422395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.2:60491)
0805 22:47:47.423010 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.423182 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":297,"threads_started":1}
I20260805 22:47:47.424535 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.424116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.2:49969)
0805 22:47:47.424298 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.424387 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260805 22:47:47.424681 14610 heartbeater.cc:503] Master 127.4.28.60:35143 was elected leader, sending a full tablet report...
I20260805 22:47:47.690929 14964 raft_consensus.cc:482] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:47.691251 14964 raft_consensus.cc:504] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } }
I20260805 22:47:47.693209 14964 leader_election.cc:288] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595), 2e0b768323044f0b8a357b08ac2054ed (127.4.28.1:33079)
I20260805 22:47:47.700254 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.693612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33079 (local address 127.4.28.2:41485)
0805 22:47:47.693936 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.693959 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:47.694266 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:47.694732 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:47.694737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:47.694763 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:47.695055 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.695069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.696316 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.696324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.698109 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.698125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.699416 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.699424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.699619 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.699662 (+ 43us) client_negotiation.cc:769] Sending connection context
0805 22:47:47.699813 (+ 151us) client_negotiation.cc:240] Negotiation successful
0805 22:47:47.699952 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260805 22:47:47.700305 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.693740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:41485 (local address 127.4.28.1:33079)
0805 22:47:47.693964 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.693971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.694072 (+ 101us) server_negotiation.cc:407] Connection header received
0805 22:47:47.694468 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.694474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.694580 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.694690 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:47.695233 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.696140 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.698290 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.699166 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.699268 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.699367 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.699866 (+ 499us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.700026 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260805 22:47:47.700683 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.693715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:57853 (local address 127.4.28.3:40595)
0805 22:47:47.693964 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.693971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.694688 (+ 717us) server_negotiation.cc:407] Connection header received
0805 22:47:47.694909 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.694915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.694995 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.695100 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:47.695767 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.696686 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.698544 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.699493 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.699645 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.699790 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.700248 (+ 458us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.700406 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260805 22:47:47.700749 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.693649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:40595 (local address 127.4.28.2:57853)
0805 22:47:47.694546 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.694567 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:47.694736 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:47.695157 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:47.695164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:47.695193 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:47.695559 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.695572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.696879 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.696883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.698356 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:47.698372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.699787 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.699794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:47.700053 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.700091 (+ 38us) client_negotiation.cc:769] Sending connection context
0805 22:47:47.700197 (+ 106us) client_negotiation.cc:240] Negotiation successful
0805 22:47:47.700329 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":3,"thread_start_us":546,"threads_started":1}
I20260805 22:47:47.701447 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75dae18db14c45bc7fb9967679985f" candidate_uuid: "2761cac4c2f84655889032c0b97978e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ed51ed787046eeba71558825437f70" is_pre_election: true
I20260805 22:47:47.701534 14326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75dae18db14c45bc7fb9967679985f" candidate_uuid: "2761cac4c2f84655889032c0b97978e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e0b768323044f0b8a357b08ac2054ed" is_pre_election: true
I20260805 22:47:47.701768 14750 raft_consensus.cc:2454] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2761cac4c2f84655889032c0b97978e1 in term 0.
I20260805 22:47:47.701954 14326 raft_consensus.cc:2454] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2761cac4c2f84655889032c0b97978e1 in term 0.
I20260805 22:47:47.702396 14446 leader_election.cc:302] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2761cac4c2f84655889032c0b97978e1, 79ed51ed787046eeba71558825437f70; no voters:
I20260805 22:47:47.702884 14964 raft_consensus.cc:2781] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:47.703034 14964 raft_consensus.cc:482] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:47.703145 14964 raft_consensus.cc:3037] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:47.705705 14964 raft_consensus.cc:504] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } }
I20260805 22:47:47.706571 14964 leader_election.cc:288] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [CANDIDATE]: Term 1 election: Requested vote from peers 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595), 2e0b768323044f0b8a357b08ac2054ed (127.4.28.1:33079)
I20260805 22:47:47.707084 14750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75dae18db14c45bc7fb9967679985f" candidate_uuid: "2761cac4c2f84655889032c0b97978e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ed51ed787046eeba71558825437f70"
I20260805 22:47:47.707151 14326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca75dae18db14c45bc7fb9967679985f" candidate_uuid: "2761cac4c2f84655889032c0b97978e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e0b768323044f0b8a357b08ac2054ed"
I20260805 22:47:47.707278 14750 raft_consensus.cc:3037] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:47.707371 14326 raft_consensus.cc:3037] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:47.709414 14750 raft_consensus.cc:2454] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2761cac4c2f84655889032c0b97978e1 in term 1.
I20260805 22:47:47.709561 14326 raft_consensus.cc:2454] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2761cac4c2f84655889032c0b97978e1 in term 1.
I20260805 22:47:47.709951 14446 leader_election.cc:302] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2761cac4c2f84655889032c0b97978e1, 79ed51ed787046eeba71558825437f70; no voters:
I20260805 22:47:47.710289 14964 raft_consensus.cc:2781] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:47.710790 14964 raft_consensus.cc:686] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 1 LEADER]: Becoming Leader. State: Replica: 2761cac4c2f84655889032c0b97978e1, State: Running, Role: LEADER
I20260805 22:47:47.711237 14964 consensus_queue.cc:237] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } }
I20260805 22:47:47.711918 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.711449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.2:43991)
0805 22:47:47.711639 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.711762 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260805 22:47:47.715024 14136 catalog_manager.cc:5495] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2761cac4c2f84655889032c0b97978e1 (127.4.28.2). New cstate: current_term: 1 leader_uuid: "2761cac4c2f84655889032c0b97978e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2761cac4c2f84655889032c0b97978e1" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 } health_report { overall_health: UNKNOWN } } }
I20260805 22:47:47.727191 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.726728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:52118)
0805 22:47:47.726878 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.727022 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260805 22:47:47.728307 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.727891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:52132)
0805 22:47:47.728094 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.728177 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260805 22:47:47.746250 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.740810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.4.28.2:42539)
0805 22:47:47.741067 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.741072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.741094 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:47:47.741306 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.741310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.741409 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.741588 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:47.741940 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.742660 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.744061 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.744489 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.744599 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.745029 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:47.745667 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:47.745778 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.745892 (+ 114us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.745972 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 22:47:47.751063 14750 raft_consensus.cc:1261] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [term 1 FOLLOWER]: Refusing update from remote peer 2761cac4c2f84655889032c0b97978e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:47.751238 14326 raft_consensus.cc:1261] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [term 1 FOLLOWER]: Refusing update from remote peer 2761cac4c2f84655889032c0b97978e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:47.751912 14968 consensus_queue.cc:1035] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ed51ed787046eeba71558825437f70" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:47.752331 14964 consensus_queue.cc:1035] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e0b768323044f0b8a357b08ac2054ed" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:47.756093 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.755600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.1:48527)
0805 22:47:47.755770 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.755879 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260805 22:47:47.756106 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.755161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.3:38129)
0805 22:47:47.755842 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.755944 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565}
I20260805 22:47:47.763103 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.762670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.3:41543)
0805 22:47:47.762810 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.762927 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260805 22:47:47.764086 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.763623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.1:34817)
0805 22:47:47.763799 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.763897 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260805 22:47:47.764222 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.763639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.4.28.2:43539)
0805 22:47:47.763928 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:47.764029 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260805 22:47:47.779659 14136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:47.779937 14136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:47.786780 14136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56b590795831486283ab1d37bc2d6120: Sending DeleteTablet for 3 replicas of tablet ca75dae18db14c45bc7fb9967679985f
I20260805 22:47:47.787863 14724 tablet_service.cc:1514] Processing DeleteTablet for tablet ca75dae18db14c45bc7fb9967679985f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:47 UTC) from {username='oryx', principal='oryx/127.4.28.60@KRBTEST.COM'} at 127.0.0.1:48444
I20260805 22:47:47.787962 14509 tablet_service.cc:1514] Processing DeleteTablet for tablet ca75dae18db14c45bc7fb9967679985f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:47 UTC) from {username='oryx', principal='oryx/127.4.28.60@KRBTEST.COM'} at 127.0.0.1:59688
I20260805 22:47:47.788321 14292 tablet_service.cc:1514] Processing DeleteTablet for tablet ca75dae18db14c45bc7fb9967679985f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:47 UTC) from {username='oryx', principal='oryx/127.4.28.60@KRBTEST.COM'} at 127.0.0.1:33506
I20260805 22:47:47.790493 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 14107
I20260805 22:47:47.802073 14987 tablet_replica.cc:330] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: stopping tablet replica
I20260805 22:47:47.802340 14987 raft_consensus.cc:2229] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:47.803649 14987 raft_consensus.cc:2258] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:47.805150 14987 ts_tablet_manager.cc:1905] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:47.809995 14986 tablet_replica.cc:330] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: stopping tablet replica
I20260805 22:47:47.810220 14986 raft_consensus.cc:2229] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:47.810966 14986 raft_consensus.cc:2258] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:47.815187 14984 tablet_replica.cc:330] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: stopping tablet replica
I20260805 22:47:47.815531 14984 raft_consensus.cc:2229] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:47.815927 14984 raft_consensus.cc:2258] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:47.817245 14984 ts_tablet_manager.cc:1905] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:47.818678 14986 ts_tablet_manager.cc:1905] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:47.824447 14984 ts_tablet_manager.cc:1918] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:47.824725 14987 ts_tablet_manager.cc:1918] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:47.824954 14987 log.cc:1196] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/ca75dae18db14c45bc7fb9967679985f
I20260805 22:47:47.825578 14987 ts_tablet_manager.cc:1939] T ca75dae18db14c45bc7fb9967679985f P 2e0b768323044f0b8a357b08ac2054ed: Deleting consensus metadata
I20260805 22:47:47.826082 14986 ts_tablet_manager.cc:1918] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:47.826336 14986 log.cc:1196] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/ca75dae18db14c45bc7fb9967679985f
I20260805 22:47:47.826385 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.792209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.4.28.1:33079)
0805 22:47:47.792422 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.792430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.797957 (+ 5527us) server_negotiation.cc:407] Connection header received
0805 22:47:47.799013 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.799021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.799120 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.799205 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:47.801945 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.803173 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.814657 (+ 11484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.815187 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.815322 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.818400 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:47.818422 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:47.818446 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:47.818499 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:47.822507 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:47.823574 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:47.823587 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:47.823591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:47.823752 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:47.824239 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:47.824245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:47.824249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:47.824772 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:47.825098 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.825737 (+ 639us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.825999 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260805 22:47:47.827225 14986 ts_tablet_manager.cc:1939] T ca75dae18db14c45bc7fb9967679985f P 2761cac4c2f84655889032c0b97978e1: Deleting consensus metadata
I20260805 22:47:47.827167 14106 catalog_manager.cc:4841] TS 2e0b768323044f0b8a357b08ac2054ed (127.4.28.1:33079): tablet ca75dae18db14c45bc7fb9967679985f (table test-table [id=3883579c827348759fb8c524b9e3243f]) successfully deleted
I20260805 22:47:47.827795 14984 log.cc:1196] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/ca75dae18db14c45bc7fb9967679985f
I20260805 22:47:47.827885 14386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:47.828470 14984 ts_tablet_manager.cc:1939] T ca75dae18db14c45bc7fb9967679985f P 79ed51ed787046eeba71558825437f70: Deleting consensus metadata
I20260805 22:47:47.828789 14104 catalog_manager.cc:4841] TS 2761cac4c2f84655889032c0b97978e1 (127.4.28.2:42539): tablet ca75dae18db14c45bc7fb9967679985f (table test-table [id=3883579c827348759fb8c524b9e3243f]) successfully deleted
I20260805 22:47:47.829842 14104 catalog_manager.cc:4841] TS 79ed51ed787046eeba71558825437f70 (127.4.28.3:40595): tablet ca75dae18db14c45bc7fb9967679985f (table test-table [id=3883579c827348759fb8c524b9e3243f]) successfully deleted
I20260805 22:47:47.951610 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 14107
I20260805 22:47:47.982262 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 14412
I20260805 22:47:47.999949 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:47.983072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.4.28.2:42539)
0805 22:47:47.983260 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:47.983265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:47.983379 (+ 114us) server_negotiation.cc:407] Connection header received
0805 22:47:47.983653 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:47.983658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:47.983738 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:47.983831 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:47.985920 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.986761 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.988762 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:47.989275 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:47.989404 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:47.993320 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:47.993353 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:47.993371 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:47.993412 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:47.996602 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:47.997606 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:47.997616 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:47.997619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:47.997810 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:47.998283 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:47.998287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:47.998289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:47.998729 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:47.999013 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:47.999406 (+ 393us) server_negotiation.cc:299] Negotiation successful
0805 22:47:47.999612 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 22:47:48.000821 14598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:48.120883 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 14412
I20260805 22:47:48.129166 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:48.128613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:52144)
0805 22:47:48.128859 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:48.128994 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260805 22:47:48.130259 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:48.129925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45685 (local address 127.0.0.1:52152)
0805 22:47:48.130063 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:48.130140 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260805 22:47:48.147239 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 14629
I20260805 22:47:48.161749 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:48.148196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.4.28.3:40595)
0805 22:47:48.148385 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:48.148391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:48.148414 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:47:48.148651 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:48.148656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:48.148747 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:48.148847 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:48.150252 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:48.151041 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:48.152253 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:48.152681 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:48.152780 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:48.155601 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:48.155622 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:48.155642 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:48.155682 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:48.158831 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:48.159599 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:48.159606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:48.159609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:48.159749 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:48.160225 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:48.160230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:48.160232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:48.160644 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:48.160890 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:48.161196 (+ 306us) server_negotiation.cc:299] Negotiation successful
0805 22:47:48.161376 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 22:47:48.162427 14805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:48.256675 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 14629
I20260805 22:47:48.282454 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 13687
I20260805 22:47:48.283697 13914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:48.400207 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 13687
I20260805 22:47:48.429325 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 13823
I20260805 22:47:48.430430 14173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:48.552913 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 13823
2026-08-05T22:47:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10628 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:48 dist-test-slave-7lbf krb5kdc[15003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:48 dist-test-slave-7lbf krb5kdc[15003](info): set up 2 sockets
Aug 05 22:47:48 dist-test-slave-7lbf krb5kdc[15003](info): commencing operation
krb5kdc: starting...
W20260805 22:47:49.754714 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:49 dist-test-slave-7lbf krb5kdc[15003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.62.keytab.
Entry for principal oryx/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.62.keytab.
I20260805 22:47:50.037204 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37155
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.62.keytab
--principal=oryx/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:37155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:50.388520 15019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:50.388850 15019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:50.388988 15019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:50.396529 15019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:50.396661 15019 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260805 22:47:50.396703 15019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:50.396776 15019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:50.396857 15019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:50.396939 15019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:50.407341 15019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:37155
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.62.keytab
--principal=oryx/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:50.408756 15019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:50.410269 15019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:50.422688 15028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:50.422693 15029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:50.422629 15032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:50.423792 15019 server_base.cc:1034] running on GCE node
I20260805 22:47:50.424643 15019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:50.426203 15019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:50.427428 15019 hybrid_clock.cc:648] HybridClock initialized: now 1785970070427360 us; error 36 us; skew 500 ppm
I20260805 22:47:50.427767 15019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:50 dist-test-slave-7lbf krb5kdc[15003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970070, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:50.434952 15019 init.cc:377] Logged in from keytab as oryx/127.4.28.62@KRBTEST.COM (short username oryx)
I20260805 22:47:50.436659 15019 webserver.cc:460] Webserver started at http://127.4.28.62:39455/ using document root <none> and password file <none>
I20260805 22:47:50.437407 15019 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:50.437542 15019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:50.437899 15019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:50.441102 15019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "2a67a65f70e9426b9720b38dfd8a21e2"
format_stamp: "Formatted at 2026-08-05 22:47:50 on dist-test-slave-7lbf"
server_key: "64ea276267d1388fdee1408720d7fc4e"
server_key_iv: "13c48e0fb506c0749c3dde87979441a2"
server_key_version: "encryptionkey@0"
I20260805 22:47:50.442032 15019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "2a67a65f70e9426b9720b38dfd8a21e2"
format_stamp: "Formatted at 2026-08-05 22:47:50 on dist-test-slave-7lbf"
server_key: "64ea276267d1388fdee1408720d7fc4e"
server_key_iv: "13c48e0fb506c0749c3dde87979441a2"
server_key_version: "encryptionkey@0"
I20260805 22:47:50.447810 15019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260805 22:47:50.451750 15041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:50.453289 15019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:47:50.453462 15019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "2a67a65f70e9426b9720b38dfd8a21e2"
format_stamp: "Formatted at 2026-08-05 22:47:50 on dist-test-slave-7lbf"
server_key: "64ea276267d1388fdee1408720d7fc4e"
server_key_iv: "13c48e0fb506c0749c3dde87979441a2"
server_key_version: "encryptionkey@0"
I20260805 22:47:50.453723 15019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:50.468816 15019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:50.473438 15019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:50.473759 15019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:50.495402 15019 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:37155
I20260805 22:47:50.495421 15121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:37155 every 8 connection(s)
I20260805 22:47:50.496910 15019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:50.500995 15123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:50.505373 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 15019
I20260805 22:47:50.505649 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:50.506465 4208 external_mini_cluster.cc:1428] Setting key 4ec00d484dfb12a5f4cb6aad0afdd664
I20260805 22:47:50.515108 15123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2: Bootstrap starting.
I20260805 22:47:50.519213 15123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:50.520713 15123 log.cc:824] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2: Log is configured to *not* fsync() on all Append() calls
Aug 05 22:47:50 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:50.523849 15123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2: No bootstrap required, opened a new log
I20260805 22:47:50.533849 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:50.508896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.4.28.62:37155)
0805 22:47:50.509942 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:50.509960 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:50.510386 (+ 426us) server_negotiation.cc:407] Connection header received
0805 22:47:50.513124 (+ 2738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:50.513159 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:50.513787 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:50.514272 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:50.515706 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:50.517789 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:50.519172 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:50.520262 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:50.520471 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:50.524684 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:50.524697 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:50.524718 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:50.524771 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:50.528806 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:50.529618 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:50.529629 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:50.529640 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:50.529879 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:50.530265 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:50.530270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:50.530273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:50.531108 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:50.531429 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:50.531744 (+ 315us) server_negotiation.cc:299] Negotiation successful
0805 22:47:50.532266 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":426,"threads_started":1}
I20260805 22:47:50.534551 15123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a67a65f70e9426b9720b38dfd8a21e2" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37155 } }
I20260805 22:47:50.534911 15123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:50.535015 15123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a67a65f70e9426b9720b38dfd8a21e2, State: Initialized, Role: FOLLOWER
I20260805 22:47:50.535599 15123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a67a65f70e9426b9720b38dfd8a21e2" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37155 } }
I20260805 22:47:50.535835 15123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:50.535967 15123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:50.536140 15123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:50.538532 15123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a67a65f70e9426b9720b38dfd8a21e2" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37155 } }
I20260805 22:47:50.539041 15123 leader_election.cc:302] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a67a65f70e9426b9720b38dfd8a21e2; no voters:
I20260805 22:47:50.539587 15123 leader_election.cc:288] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:50.539728 15131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:50.540863 15131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [term 1 LEADER]: Becoming Leader. State: Replica: 2a67a65f70e9426b9720b38dfd8a21e2, State: Running, Role: LEADER
I20260805 22:47:50.541477 15131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a67a65f70e9426b9720b38dfd8a21e2" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37155 } }
I20260805 22:47:50.541797 15123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:50.544337 15132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a67a65f70e9426b9720b38dfd8a21e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a67a65f70e9426b9720b38dfd8a21e2" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37155 } } }
I20260805 22:47:50.544603 15133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a67a65f70e9426b9720b38dfd8a21e2. Latest consensus state: current_term: 1 leader_uuid: "2a67a65f70e9426b9720b38dfd8a21e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a67a65f70e9426b9720b38dfd8a21e2" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37155 } } }
I20260805 22:47:50.544859 15132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:50.544898 15133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2 [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:50.546773 15144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:50.551936 15144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:50.562170 15144 catalog_manager.cc:1347] Generated new cluster ID: d8322fe2b21945a0aa209acb5d05b96f
I20260805 22:47:50.562309 15144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:50.580519 15144 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:50.582072 15144 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:50.593451 15144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2: Generated new TSK 0
I20260805 22:47:50.594442 15144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab.
Entry for principal oryx/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab.
I20260805 22:47:50.663558 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37155
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab
--principal=oryx/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:51.023038 15165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:51.023342 15165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:51.023470 15165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:51.031003 15165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:51.031127 15165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:51.031234 15165 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 22:47:51.031345 15165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:51.041754 15165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab
--principal=oryx/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.28.62:37155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:51.043161 15165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:51.044670 15165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:51.058873 15173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:51.058856 15174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:51.059087 15176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:51.060138 15165 server_base.cc:1034] running on GCE node
I20260805 22:47:51.060809 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:51.061901 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:51.063143 15165 hybrid_clock.cc:648] HybridClock initialized: now 1785970071063064 us; error 45 us; skew 500 ppm
I20260805 22:47:51.063476 15165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:51 dist-test-slave-7lbf krb5kdc[15003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970071, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:51.066819 15165 init.cc:377] Logged in from keytab as oryx/127.4.28.1@KRBTEST.COM (short username oryx)
I20260805 22:47:51.068382 15165 webserver.cc:460] Webserver started at http://127.4.28.1:36225/ using document root <none> and password file <none>
I20260805 22:47:51.069131 15165 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:51.069236 15165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:51.069608 15165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:51.072799 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "11aedee574e043d1853bb259393b5c22"
format_stamp: "Formatted at 2026-08-05 22:47:51 on dist-test-slave-7lbf"
server_key: "43c264b8535e76167b70d578fa46604a"
server_key_iv: "677f4567d581cc983f0b66d981f3a6c0"
server_key_version: "encryptionkey@0"
I20260805 22:47:51.073720 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "11aedee574e043d1853bb259393b5c22"
format_stamp: "Formatted at 2026-08-05 22:47:51 on dist-test-slave-7lbf"
server_key: "43c264b8535e76167b70d578fa46604a"
server_key_iv: "677f4567d581cc983f0b66d981f3a6c0"
server_key_version: "encryptionkey@0"
I20260805 22:47:51.079380 15165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260805 22:47:51.083315 15189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:51.084862 15165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 22:47:51.085049 15165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "11aedee574e043d1853bb259393b5c22"
format_stamp: "Formatted at 2026-08-05 22:47:51 on dist-test-slave-7lbf"
server_key: "43c264b8535e76167b70d578fa46604a"
server_key_iv: "677f4567d581cc983f0b66d981f3a6c0"
server_key_version: "encryptionkey@0"
I20260805 22:47:51.085282 15165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:51.108174 15165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:51.113107 15165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:51.113413 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:51.116482 15165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:51.116601 15165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:51.116740 15165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:51.116809 15165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:47:51 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970071, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:51.142953 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:51.117392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:57095 (local address 127.4.28.62:37155)
0805 22:47:51.117905 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:51.117910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:51.119566 (+ 1656us) server_negotiation.cc:407] Connection header received
0805 22:47:51.121846 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:51.121853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:51.121980 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:51.122114 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:51.124792 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.126029 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.128416 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.129144 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.129297 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:51.135349 (+ 6052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:51.135358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:51.135363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:51.135418 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:51.138603 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:51.139760 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:51.139767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:51.139771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:51.139945 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:51.140589 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:51.140593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:51.140595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:51.140969 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:51.141141 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:51.142273 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0805 22:47:51.142538 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":277,"threads_started":1}
I20260805 22:47:51.144243 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:51.118246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37155 (local address 127.4.28.1:57095)
0805 22:47:51.119365 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:51.119421 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:51.121411 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:51.122567 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:51.122576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:51.123502 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:51.124505 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:51.124548 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.126250 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.126257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:51.128209 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:51.128230 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.129760 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.129770 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:51.130141 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:51.131116 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:51.131165 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:51.134938 (+ 3773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:51.138831 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:51.138842 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:51.138860 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:51.139544 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:51.140073 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:51.140078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:51.140082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:51.140415 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:51.141216 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:51.141225 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:51.141831 (+ 606us) client_negotiation.cc:769] Sending connection context
0805 22:47:51.142134 (+ 303us) client_negotiation.cc:240] Negotiation successful
0805 22:47:51.142758 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":474,"threads_started":1}
I20260805 22:47:51.166227 15165 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:46761
I20260805 22:47:51.166304 15353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:46761 every 8 connection(s)
I20260805 22:47:51.168002 15165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:51.170639 15354 heartbeater.cc:347] Connected to a master server at 127.4.28.62:37155
I20260805 22:47:51.170928 15354 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:51.171694 15354 heartbeater.cc:511] Master 127.4.28.62:37155 requested a full tablet report, sending...
I20260805 22:47:51.173485 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 15165
I20260805 22:47:51.173739 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:51.174089 15062 ts_manager.cc:194] Registered new tserver with Master: 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761)
I20260805 22:47:51.174451 4208 external_mini_cluster.cc:1428] Setting key 69e84e9279745c3c515aff52d06c4a60
I20260805 22:47:51.175918 15062 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:57095
I20260805 22:47:51.176662 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:51.160947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37155 (local address 127.4.28.1:38123)
0805 22:47:51.161869 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:51.161894 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:51.162100 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:51.162531 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:51.162535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:51.163038 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:51.163364 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:51.163376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.164755 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.164761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:51.166530 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:51.166546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.167635 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.167641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:51.167783 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:51.168507 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:51.168534 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:51.169775 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:51.173560 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:51.173566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:51.173571 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:51.174107 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:51.174659 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:51.174664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:51.174667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:51.174824 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:51.175494 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:51.175498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:51.175721 (+ 223us) client_negotiation.cc:769] Sending connection context
0805 22:47:51.175868 (+ 147us) client_negotiation.cc:240] Negotiation successful
0805 22:47:51.176228 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":483,"threads_started":1}
I20260805 22:47:51.177261 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:51.160548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:38123 (local address 127.4.28.62:37155)
0805 22:47:51.160775 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:51.160783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:51.162017 (+ 1234us) server_negotiation.cc:407] Connection header received
0805 22:47:51.162260 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:51.162268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:51.162366 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:51.162492 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:51.163525 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.164580 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.166690 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.167356 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.167518 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:51.170021 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:51.170030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:51.170035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:51.170084 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:51.173350 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:51.174316 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:51.174332 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:51.174336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:51.174530 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:51.174956 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:51.174961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:51.174964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:51.175372 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:51.176561 (+ 1189us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:51.176717 (+ 156us) server_negotiation.cc:299] Negotiation successful
0805 22:47:51.176962 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for oryx/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab.
Entry for principal oryx/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab.
I20260805 22:47:51.236644 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37155
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab
--principal=oryx/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:51.590634 15359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:51.590935 15359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:51.591037 15359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:51.598453 15359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:51.598569 15359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:51.598667 15359 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 22:47:51.598749 15359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:51.608827 15359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab
--principal=oryx/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.28.62:37155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:51.610195 15359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:51.611640 15359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:51.625538 15369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:51.625583 15367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:51.625766 15366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:51.626739 15359 server_base.cc:1034] running on GCE node
I20260805 22:47:51.627388 15359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:51.628351 15359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:51.629616 15359 hybrid_clock.cc:648] HybridClock initialized: now 1785970071629517 us; error 60 us; skew 500 ppm
I20260805 22:47:51.629930 15359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:51 dist-test-slave-7lbf krb5kdc[15003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970071, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:51.633325 15359 init.cc:377] Logged in from keytab as oryx/127.4.28.2@KRBTEST.COM (short username oryx)
I20260805 22:47:51.635079 15359 webserver.cc:460] Webserver started at http://127.4.28.2:41241/ using document root <none> and password file <none>
I20260805 22:47:51.635834 15359 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:51.635948 15359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:51.636300 15359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:51.639564 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "9a7a5aab572240948df5ed5c977d54b9"
format_stamp: "Formatted at 2026-08-05 22:47:51 on dist-test-slave-7lbf"
server_key: "1c163307e242a47e40e4bfaa0826b98e"
server_key_iv: "7b4c3882d3673a31d6267cb0bc43ab24"
server_key_version: "encryptionkey@0"
I20260805 22:47:51.640410 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "9a7a5aab572240948df5ed5c977d54b9"
format_stamp: "Formatted at 2026-08-05 22:47:51 on dist-test-slave-7lbf"
server_key: "1c163307e242a47e40e4bfaa0826b98e"
server_key_iv: "7b4c3882d3673a31d6267cb0bc43ab24"
server_key_version: "encryptionkey@0"
I20260805 22:47:51.646349 15359 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260805 22:47:51.650481 15377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:51.652024 15359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:51.652210 15359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "9a7a5aab572240948df5ed5c977d54b9"
format_stamp: "Formatted at 2026-08-05 22:47:51 on dist-test-slave-7lbf"
server_key: "1c163307e242a47e40e4bfaa0826b98e"
server_key_iv: "7b4c3882d3673a31d6267cb0bc43ab24"
server_key_version: "encryptionkey@0"
I20260805 22:47:51.652442 15359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:51.676008 15359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:51.680701 15359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:51.680982 15359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:51.683840 15359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:51.683940 15359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:51.684070 15359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:51.684130 15359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:47:51 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970071, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:51.710256 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:51.684537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:58239 (local address 127.4.28.62:37155)
0805 22:47:51.685059 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:51.685066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:51.686416 (+ 1350us) server_negotiation.cc:407] Connection header received
0805 22:47:51.689663 (+ 3247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:51.689670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:51.689789 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:51.689927 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:51.692509 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.693584 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.696647 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.697208 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.697351 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:51.702823 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:51.702829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:51.702833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:51.702875 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:51.705649 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:51.706745 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:51.706753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:51.706758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:51.706958 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:51.707631 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:51.707636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:51.707639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:51.708065 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:51.708228 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:51.709501 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0805 22:47:51.709846 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":266,"threads_started":1}
I20260805 22:47:51.712504 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:51.685253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37155 (local address 127.4.28.2:58239)
0805 22:47:51.686210 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:51.686276 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:51.689187 (+ 2911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:51.690264 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:51.690277 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:51.691216 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:51.692205 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:51.692237 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.693763 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.693769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:51.696430 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:51.696455 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.697724 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.697732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:51.698035 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:51.698911 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:51.698942 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:51.702534 (+ 3592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:51.705829 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:51.705840 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:51.705862 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:51.706578 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:51.707128 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:51.707134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:51.707139 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:51.707488 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:51.708251 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:51.708262 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:51.708911 (+ 649us) client_negotiation.cc:769] Sending connection context
0805 22:47:51.709389 (+ 478us) client_negotiation.cc:240] Negotiation successful
0805 22:47:51.710132 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":376,"threads_started":1}
I20260805 22:47:51.727648 15359 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:42917
I20260805 22:47:51.727660 15556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:42917 every 8 connection(s)
I20260805 22:47:51.729450 15359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:47:51.731886 15557 heartbeater.cc:347] Connected to a master server at 127.4.28.62:37155
I20260805 22:47:51.732141 15557 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:51.733002 15557 heartbeater.cc:511] Master 127.4.28.62:37155 requested a full tablet report, sending...
I20260805 22:47:51.734716 15062 ts_manager.cc:194] Registered new tserver with Master: 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917)
I20260805 22:47:51.735682 15062 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:58239
I20260805 22:47:51.736229 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 15359
I20260805 22:47:51.736439 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:47:51.736951 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:51.720009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37155 (local address 127.4.28.2:60455)
0805 22:47:51.722044 (+ 2035us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:51.722065 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:51.722234 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:51.722680 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:51.722687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:51.723605 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:51.724143 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:51.724168 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.725501 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.725507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:51.726914 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:51.726933 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.728138 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.728145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:51.728327 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:51.729060 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:51.729084 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:51.730296 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:51.734039 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:51.734044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:51.734047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:51.734556 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:51.735060 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:51.735065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:51.735068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:51.735232 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:51.735876 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:51.735881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:51.736085 (+ 204us) client_negotiation.cc:769] Sending connection context
0805 22:47:51.736226 (+ 141us) client_negotiation.cc:240] Negotiation successful
0805 22:47:51.736613 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1879,"thread_start_us":1707,"threads_started":1}
I20260805 22:47:51.737025 4208 external_mini_cluster.cc:1428] Setting key 363c192dc8688e546ace9580220c93a4
I20260805 22:47:51.737735 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:51.720062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:60455 (local address 127.4.28.62:37155)
0805 22:47:51.720284 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:51.720291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:51.722175 (+ 1884us) server_negotiation.cc:407] Connection header received
0805 22:47:51.722370 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:51.722376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:51.722467 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:51.722570 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:51.724370 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.725299 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.727075 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:51.727722 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:51.727862 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:51.730569 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:51.730574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:51.730577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:51.730612 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:51.733889 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:51.734718 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:51.734723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:51.734725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:51.734928 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:51.735332 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:51.735335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:51.735337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:51.735712 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:51.735887 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:51.737260 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0805 22:47:51.737438 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for oryx/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab.
Entry for principal oryx/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab.
I20260805 22:47:51.804703 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37155
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab
--principal=oryx/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:52.157047 15563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:52.157397 15563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:52.157583 15563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:52.165324 15563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:52.165501 15563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:52.165668 15563 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 22:47:52.165800 15563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:47:52.178681 15563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab
--principal=oryx/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.28.62:37155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:52.179744 15354 heartbeater.cc:503] Master 127.4.28.62:37155 was elected leader, sending a full tablet report...
I20260805 22:47:52.180045 15563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:52.181715 15563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:52.195544 15579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:52.195623 15577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:52.195623 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:52.196733 15563 server_base.cc:1034] running on GCE node
I20260805 22:47:52.197461 15563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:52.198539 15563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:52.199815 15563 hybrid_clock.cc:648] HybridClock initialized: now 1785970072199736 us; error 52 us; skew 500 ppm
I20260805 22:47:52.200169 15563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:52 dist-test-slave-7lbf krb5kdc[15003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970072, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:52.203650 15563 init.cc:377] Logged in from keytab as oryx/127.4.28.3@KRBTEST.COM (short username oryx)
I20260805 22:47:52.205407 15563 webserver.cc:460] Webserver started at http://127.4.28.3:41207/ using document root <none> and password file <none>
I20260805 22:47:52.206204 15563 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:52.206321 15563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:52.206656 15563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:52.209932 15563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "419750b94684451085a90fe588f31632"
format_stamp: "Formatted at 2026-08-05 22:47:52 on dist-test-slave-7lbf"
server_key: "82397a13ac9b246d79f03ee58ec4e0fc"
server_key_iv: "aca02aa4b0a1a60ce36a3b58f0ae767a"
server_key_version: "encryptionkey@0"
I20260805 22:47:52.210811 15563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "419750b94684451085a90fe588f31632"
format_stamp: "Formatted at 2026-08-05 22:47:52 on dist-test-slave-7lbf"
server_key: "82397a13ac9b246d79f03ee58ec4e0fc"
server_key_iv: "aca02aa4b0a1a60ce36a3b58f0ae767a"
server_key_version: "encryptionkey@0"
I20260805 22:47:52.216475 15563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:47:52.220614 15589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:52.222237 15563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:47:52.222415 15563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "419750b94684451085a90fe588f31632"
format_stamp: "Formatted at 2026-08-05 22:47:52 on dist-test-slave-7lbf"
server_key: "82397a13ac9b246d79f03ee58ec4e0fc"
server_key_iv: "aca02aa4b0a1a60ce36a3b58f0ae767a"
server_key_version: "encryptionkey@0"
I20260805 22:47:52.222654 15563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:52.239265 15563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:52.243892 15563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:52.244165 15563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:52.247226 15563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:52.247329 15563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:52.247541 15563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:52.247612 15563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:47:52 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970072, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:52.273341 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.247787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:50937 (local address 127.4.28.62:37155)
0805 22:47:52.248572 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.248578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.249733 (+ 1155us) server_negotiation.cc:407] Connection header received
0805 22:47:52.251956 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.251963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.252081 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.252202 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:52.255239 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.256500 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.259796 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.260406 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.260563 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.265272 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:52.265278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:52.265282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:52.265324 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:52.268958 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:52.270467 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:52.270475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:52.270479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:52.270653 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:52.271195 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:52.271202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:52.271205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:52.271602 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:52.271755 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.272639 (+ 884us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.272893 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":527,"threads_started":1}
I20260805 22:47:52.274448 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.248557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37155 (local address 127.4.28.3:50937)
0805 22:47:52.249504 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:52.249594 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:52.251625 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:52.252668 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:52.252683 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:52.253809 (+ 1126us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:52.254926 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.254962 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.256761 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.256770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.259537 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.259564 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.260703 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.260712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.260962 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.261857 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:52.261899 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:52.264960 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:52.269229 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:52.269242 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:52.269268 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:52.270242 (+ 974us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:52.270803 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:52.270807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:52.270810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:52.271047 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:52.271754 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:52.271764 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:52.272185 (+ 421us) client_negotiation.cc:769] Sending connection context
0805 22:47:52.272450 (+ 265us) client_negotiation.cc:240] Negotiation successful
0805 22:47:52.273072 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":385,"threads_started":1}
I20260805 22:47:52.299036 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.284361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:38355 (local address 127.4.28.62:37155)
0805 22:47:52.284565 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.284572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.284833 (+ 261us) server_negotiation.cc:407] Connection header received
0805 22:47:52.285067 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.285071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.285146 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.285244 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:52.286764 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.287587 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.289220 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.289989 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.290157 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.291988 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:52.291995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:52.292001 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:52.292044 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:52.295754 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:52.296678 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:52.296684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:52.296687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:52.296862 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:52.297237 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:52.297242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:52.297245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:52.297692 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:52.297853 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.298415 (+ 562us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.298658 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260805 22:47:52.299132 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.283935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37155 (local address 127.4.28.3:38355)
0805 22:47:52.284686 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:52.284715 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:52.284913 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:52.285298 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:52.285303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:52.286039 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:52.286583 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.286598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.287759 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.287765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.289036 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.289059 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.290221 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.290229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.290356 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.290979 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:52.290997 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:52.291762 (+ 765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:52.295933 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:52.295947 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:52.295950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:52.296532 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:52.296986 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:52.296990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:52.296992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:52.297135 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:52.297842 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:52.297846 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:52.298129 (+ 283us) client_negotiation.cc:769] Sending connection context
0805 22:47:52.298298 (+ 169us) client_negotiation.cc:240] Negotiation successful
0805 22:47:52.298670 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":508,"threads_started":1}
I20260805 22:47:52.308046 15563 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:36099
I20260805 22:47:52.308092 15748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:36099 every 8 connection(s)
I20260805 22:47:52.309338 15563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:47:52.311685 15749 heartbeater.cc:347] Connected to a master server at 127.4.28.62:37155
I20260805 22:47:52.311949 15749 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:52.312697 15749 heartbeater.cc:511] Master 127.4.28.62:37155 requested a full tablet report, sending...
I20260805 22:47:52.314354 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 15563
I20260805 22:47:52.314453 15060 ts_manager.cc:194] Registered new tserver with Master: 419750b94684451085a90fe588f31632 (127.4.28.3:36099)
I20260805 22:47:52.314595 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:47:52.315160 4208 external_mini_cluster.cc:1428] Setting key a813503986b10e4753da14cfa4eecad6
I20260805 22:47:52.315780 15060 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:50937
I20260805 22:47:52.318549 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 22:47:52 dist-test-slave-7lbf krb5kdc[15003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 05 22:47:52 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:47:52 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
W20260805 22:47:52.380959 15599 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:52.369662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.4.28.62:37155)
0805 22:47:52.369916 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.369923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.370101 (+ 178us) server_negotiation.cc:407] Connection header received
0805 22:47:52.370286 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.370290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.370360 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.370460 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:52.371860 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.372631 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.374120 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.374549 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.374652 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.380711 (+ 6059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53886: BlockingRecv error: recv got EOF from 127.0.0.1:53886 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 05 22:47:52 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:52.399318 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.384586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.4.28.62:37155)
0805 22:47:52.384798 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.384805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.385063 (+ 258us) server_negotiation.cc:407] Connection header received
0805 22:47:52.385276 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.385280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.385352 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.385439 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:52.386814 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.387574 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.388907 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.389584 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.389734 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.393860 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:52.393867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:52.393871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:52.393921 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:52.396569 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:52.397340 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:52.397345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:52.397348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:52.397567 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:52.398013 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:52.398017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:52.398020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:52.398336 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:52.398572 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.398858 (+ 286us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.399065 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 22:47:52.404335 15060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:52.411675 15060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:52.435405 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.425209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.4.28.2:42917)
0805 22:47:52.426846 (+ 1637us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.426857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.426890 (+ 33us) server_negotiation.cc:407] Connection header received
0805 22:47:52.427011 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.427019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.427302 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.427574 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:52.428337 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.429823 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.432773 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.434213 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.434346 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.434573 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.434883 (+ 310us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.435066 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1484,"thread_start_us":1343,"threads_started":1}
I20260805 22:47:52.435475 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.424994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42917 (local address 127.0.0.1:58840)
0805 22:47:52.426010 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:52.426065 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:52.426248 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:52.427570 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:52.427583 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:52.427614 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:52.428085 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.428102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.430012 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.430017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.432547 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.432578 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.434411 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.434419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.434616 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.434678 (+ 62us) client_negotiation.cc:769] Sending connection context
0805 22:47:52.434839 (+ 161us) client_negotiation.cc:240] Negotiation successful
0805 22:47:52.435048 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":1,"thread_start_us":568,"threads_started":1}
I20260805 22:47:52.437278 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.425149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:36099 (local address 127.0.0.1:32916)
0805 22:47:52.427700 (+ 2551us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:52.427718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:52.427849 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:52.429762 (+ 1913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:52.429767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:52.429793 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:52.430230 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.430241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.432022 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.432027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.433890 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.433905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.436483 (+ 2578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.436489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.436654 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.436688 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 22:47:52.436769 (+ 81us) client_negotiation.cc:240] Negotiation successful
0805 22:47:52.436914 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2430,"thread_start_us":2295,"threads_started":1}
I20260805 22:47:52.437340 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.425889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.4.28.3:36099)
0805 22:47:52.427293 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.427301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.428648 (+ 1347us) server_negotiation.cc:407] Connection header received
0805 22:47:52.428753 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.428760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.429028 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.429200 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:52.430432 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.431853 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.434046 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.435383 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.436066 (+ 683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.436336 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.436873 (+ 537us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.437038 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1236,"thread_start_us":1044,"threads_started":1}
I20260805 22:47:52.437704 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.427866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:46761 (local address 127.0.0.1:36870)
0805 22:47:52.428398 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:52.428417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:52.428518 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:52.430765 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:52.430769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:52.430799 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:52.431266 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.431279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.433096 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.433102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.435090 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.435105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.436935 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.436942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.437132 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.437168 (+ 36us) client_negotiation.cc:769] Sending connection context
0805 22:47:52.437234 (+ 66us) client_negotiation.cc:240] Negotiation successful
0805 22:47:52.437365 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":333,"threads_started":1}
I20260805 22:47:52.438398 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.428211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.4.28.1:46761)
0805 22:47:52.430155 (+ 1944us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.430164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.430192 (+ 28us) server_negotiation.cc:407] Connection header received
0805 22:47:52.430290 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.430299 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.430559 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.430756 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:52.431483 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.432924 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.435283 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.436693 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.437639 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.437873 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.437947 (+ 74us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.438116 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1775,"thread_start_us":1620,"threads_started":1}
I20260805 22:47:52.440145 15445 tablet_service.cc:1467] Processing CreateTablet for tablet 805a9afab3614df3937a769b3d7600c1 (DEFAULT_TABLE table=test-table [id=be084e5696834617a999d4f44ce5acc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:52.441910 15445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805a9afab3614df3937a769b3d7600c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:52.441414 15652 tablet_service.cc:1467] Processing CreateTablet for tablet 805a9afab3614df3937a769b3d7600c1 (DEFAULT_TABLE table=test-table [id=be084e5696834617a999d4f44ce5acc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:52.441677 15257 tablet_service.cc:1467] Processing CreateTablet for tablet 805a9afab3614df3937a769b3d7600c1 (DEFAULT_TABLE table=test-table [id=be084e5696834617a999d4f44ce5acc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:52.442922 15257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805a9afab3614df3937a769b3d7600c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:52.442952 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805a9afab3614df3937a769b3d7600c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:52.454988 15786 tablet_bootstrap.cc:492] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: Bootstrap starting.
I20260805 22:47:52.457798 15787 tablet_bootstrap.cc:492] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: Bootstrap starting.
I20260805 22:47:52.459493 15788 tablet_bootstrap.cc:492] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: Bootstrap starting.
I20260805 22:47:52.460943 15786 tablet_bootstrap.cc:654] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:52.461400 15787 tablet_bootstrap.cc:654] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:52.462433 15787 log.cc:824] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:52.462433 15786 log.cc:824] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:52.463275 15788 tablet_bootstrap.cc:654] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:52.464421 15788 log.cc:824] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:52.465605 15787 tablet_bootstrap.cc:492] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: No bootstrap required, opened a new log
I20260805 22:47:52.465898 15787 ts_tablet_manager.cc:1397] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260805 22:47:52.466300 15786 tablet_bootstrap.cc:492] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: No bootstrap required, opened a new log
I20260805 22:47:52.466583 15786 ts_tablet_manager.cc:1397] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260805 22:47:52.467579 15788 tablet_bootstrap.cc:492] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: No bootstrap required, opened a new log
I20260805 22:47:52.467857 15788 ts_tablet_manager.cc:1397] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260805 22:47:52.473656 15787 raft_consensus.cc:348] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.474076 15787 raft_consensus.cc:374] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:52.474200 15787 raft_consensus.cc:729] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7a5aab572240948df5ed5c977d54b9, State: Initialized, Role: FOLLOWER
I20260805 22:47:52.474838 15787 consensus_queue.cc:260] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.476011 15557 heartbeater.cc:503] Master 127.4.28.62:37155 was elected leader, sending a full tablet report...
I20260805 22:47:52.476728 15787 ts_tablet_manager.cc:1428] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260805 22:47:52.477425 15786 raft_consensus.cc:348] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.477952 15786 raft_consensus.cc:374] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:52.478078 15786 raft_consensus.cc:729] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11aedee574e043d1853bb259393b5c22, State: Initialized, Role: FOLLOWER
I20260805 22:47:52.478408 15788 raft_consensus.cc:348] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.478881 15788 raft_consensus.cc:374] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:52.478753 15786 consensus_queue.cc:260] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.478986 15788 raft_consensus.cc:729] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419750b94684451085a90fe588f31632, State: Initialized, Role: FOLLOWER
I20260805 22:47:52.479735 15788 consensus_queue.cc:260] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.481489 15786 ts_tablet_manager.cc:1428] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260805 22:47:52.481638 15558 tablet.cc:2369] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:47:52.483026 15749 heartbeater.cc:503] Master 127.4.28.62:37155 was elected leader, sending a full tablet report...
I20260805 22:47:52.484005 15788 ts_tablet_manager.cc:1428] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260805 22:47:52.561020 15751 tablet.cc:2369] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:47:52.672042 15356 tablet.cc:2369] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:47:52.743695 15797 raft_consensus.cc:482] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:52.744027 15797 raft_consensus.cc:504] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.745764 15797 leader_election.cc:288] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917), 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761)
I20260805 22:47:52.746780 15793 raft_consensus.cc:482] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:52.747103 15793 raft_consensus.cc:504] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.749293 15793 leader_election.cc:288] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 419750b94684451085a90fe588f31632 (127.4.28.3:36099), 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761)
I20260805 22:47:52.754298 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.746161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42917 (local address 127.4.28.3:39437)
0805 22:47:52.746532 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:52.746555 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:52.746709 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:52.747376 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:52.747382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:52.747475 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:52.747718 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.747734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.749225 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.749231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.751508 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.751525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.753399 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.753405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.753655 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.753695 (+ 40us) client_negotiation.cc:769] Sending connection context
0805 22:47:52.753789 (+ 94us) client_negotiation.cc:240] Negotiation successful
0805 22:47:52.753949 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":96,"spinlock_wait_cycles":27904}
I20260805 22:47:52.755223 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.746306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:39437 (local address 127.4.28.2:42917)
0805 22:47:52.746794 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.746800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.746821 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:47:52.746904 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.746909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.747055 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.747171 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:52.747902 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.749054 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.751712 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.753062 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.753195 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.753306 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.754777 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.754937 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355}
I20260805 22:47:52.755631 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.746329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:55521 (local address 127.4.28.1:46761)
0805 22:47:52.746572 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.746578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.747398 (+ 820us) server_negotiation.cc:407] Connection header received
0805 22:47:52.747598 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.747603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.747719 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.747809 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:52.748335 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.749053 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.752990 (+ 3937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.754315 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.754544 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.754654 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.755180 (+ 526us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.755344 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":74}
I20260805 22:47:52.756422 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.746257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:46761 (local address 127.4.28.3:55521)
0805 22:47:52.747290 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:52.747306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:52.747432 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:52.747840 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:52.747844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:52.747881 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:52.748161 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.748169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.749201 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.749206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.752792 (+ 3586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.752812 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.754788 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.754795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.754975 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.755011 (+ 36us) client_negotiation.cc:769] Sending connection context
0805 22:47:52.755807 (+ 796us) client_negotiation.cc:240] Negotiation successful
0805 22:47:52.755963 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":29,"thread_start_us":751,"threads_started":1}
I20260805 22:47:52.756573 15474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805a9afab3614df3937a769b3d7600c1" candidate_uuid: "419750b94684451085a90fe588f31632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7a5aab572240948df5ed5c977d54b9" is_pre_election: true
I20260805 22:47:52.757165 15474 raft_consensus.cc:2454] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 419750b94684451085a90fe588f31632 in term 0.
I20260805 22:47:52.758167 15284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805a9afab3614df3937a769b3d7600c1" candidate_uuid: "419750b94684451085a90fe588f31632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22" is_pre_election: true
I20260805 22:47:52.758646 15590 leader_election.cc:302] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419750b94684451085a90fe588f31632, 9a7a5aab572240948df5ed5c977d54b9; no voters:
I20260805 22:47:52.758832 15284 raft_consensus.cc:2454] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 419750b94684451085a90fe588f31632 in term 0.
I20260805 22:47:52.759269 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.749598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:36099 (local address 127.4.28.2:44491)
0805 22:47:52.750505 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:52.750527 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:52.750743 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:52.751257 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:52.751261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:52.751288 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:52.752556 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.752568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.754272 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.754277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.756514 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.756532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.758301 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.758309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.758591 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.758630 (+ 39us) client_negotiation.cc:769] Sending connection context
0805 22:47:52.758729 (+ 99us) client_negotiation.cc:240] Negotiation successful
0805 22:47:52.758889 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":66,"thread_start_us":550,"threads_started":1}
I20260805 22:47:52.759239 15797 raft_consensus.cc:2781] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:52.759557 15797 raft_consensus.cc:482] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:52.759897 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.749783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:44491 (local address 127.4.28.3:36099)
0805 22:47:52.750016 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.750022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.750629 (+ 607us) server_negotiation.cc:407] Connection header received
0805 22:47:52.750880 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.750886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.750971 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.751033 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:52.753003 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.754088 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.756720 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.757899 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.758019 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.758128 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.759480 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.759630 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260805 22:47:52.759721 15797 raft_consensus.cc:3037] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:52.760694 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805a9afab3614df3937a769b3d7600c1" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419750b94684451085a90fe588f31632" is_pre_election: true
I20260805 22:47:52.761204 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.750679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:46761 (local address 127.4.28.2:44619)
0805 22:47:52.752113 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:52.752132 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:52.752255 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:52.753564 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:52.753569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:52.753598 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:52.753968 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.753980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.756357 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.756363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.758561 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:52.758579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.760344 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.760353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:52.760530 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.760567 (+ 37us) client_negotiation.cc:769] Sending connection context
0805 22:47:52.760671 (+ 104us) client_negotiation.cc:240] Negotiation successful
0805 22:47:52.760818 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"mutex_wait_us":1,"thread_start_us":1118,"threads_started":1}
I20260805 22:47:52.761212 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.751151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:44619 (local address 127.4.28.1:46761)
0805 22:47:52.753193 (+ 2042us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.753199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.753221 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:47:52.753281 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.753287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.753371 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.754127 (+ 756us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:52.754452 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.756205 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.758778 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.760066 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.760251 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.760377 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.760724 (+ 347us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.760880 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":512,"server-negotiator.queue_time_us":1888,"thread_start_us":1612,"threads_started":1}
I20260805 22:47:52.761860 15284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805a9afab3614df3937a769b3d7600c1" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22" is_pre_election: true
I20260805 22:47:52.762076 15284 raft_consensus.cc:2454] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7a5aab572240948df5ed5c977d54b9 in term 0.
I20260805 22:47:52.762578 15381 leader_election.cc:302] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aedee574e043d1853bb259393b5c22, 9a7a5aab572240948df5ed5c977d54b9; no voters:
I20260805 22:47:52.762961 15793 raft_consensus.cc:2781] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:52.763108 15793 raft_consensus.cc:482] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:52.763209 15793 raft_consensus.cc:3037] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:52.763101 15797 raft_consensus.cc:504] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.763640 15685 raft_consensus.cc:2379] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a7a5aab572240948df5ed5c977d54b9 in current term 1: Already voted for candidate 419750b94684451085a90fe588f31632 in this term.
I20260805 22:47:52.764070 15797 leader_election.cc:288] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [CANDIDATE]: Term 1 election: Requested vote from peers 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917), 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761)
I20260805 22:47:52.764690 15284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805a9afab3614df3937a769b3d7600c1" candidate_uuid: "419750b94684451085a90fe588f31632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22"
I20260805 22:47:52.764743 15474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805a9afab3614df3937a769b3d7600c1" candidate_uuid: "419750b94684451085a90fe588f31632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7a5aab572240948df5ed5c977d54b9"
I20260805 22:47:52.764910 15284 raft_consensus.cc:3037] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:52.766438 15793 raft_consensus.cc:504] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.766887 15474 raft_consensus.cc:2379] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 419750b94684451085a90fe588f31632 in current term 1: Already voted for candidate 9a7a5aab572240948df5ed5c977d54b9 in this term.
I20260805 22:47:52.767295 15793 leader_election.cc:288] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 election: Requested vote from peers 419750b94684451085a90fe588f31632 (127.4.28.3:36099), 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761)
I20260805 22:47:52.767701 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805a9afab3614df3937a769b3d7600c1" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419750b94684451085a90fe588f31632"
I20260805 22:47:52.767975 15685 raft_consensus.cc:2379] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a7a5aab572240948df5ed5c977d54b9 in current term 1: Already voted for candidate 419750b94684451085a90fe588f31632 in this term.
I20260805 22:47:52.768002 15284 raft_consensus.cc:2454] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 419750b94684451085a90fe588f31632 in term 1.
I20260805 22:47:52.768440 15591 leader_election.cc:302] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aedee574e043d1853bb259393b5c22, 419750b94684451085a90fe588f31632; no voters:
I20260805 22:47:52.768682 15284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805a9afab3614df3937a769b3d7600c1" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22"
I20260805 22:47:52.768754 15797 raft_consensus.cc:2781] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:52.768975 15284 raft_consensus.cc:2379] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a7a5aab572240948df5ed5c977d54b9 in current term 1: Already voted for candidate 419750b94684451085a90fe588f31632 in this term.
I20260805 22:47:52.769245 15797 raft_consensus.cc:686] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 1 LEADER]: Becoming Leader. State: Replica: 419750b94684451085a90fe588f31632, State: Running, Role: LEADER
I20260805 22:47:52.769749 15381 leader_election.cc:302] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a7a5aab572240948df5ed5c977d54b9; no voters: 11aedee574e043d1853bb259393b5c22, 419750b94684451085a90fe588f31632
I20260805 22:47:52.769802 15797 consensus_queue.cc:237] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } }
I20260805 22:47:52.770032 15793 raft_consensus.cc:2726] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:47:52.774163 15060 catalog_manager.cc:5495] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 reported cstate change: term changed from 0 to 1, leader changed from <none> to 419750b94684451085a90fe588f31632 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "419750b94684451085a90fe588f31632" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } health_report { overall_health: UNKNOWN } } }
I20260805 22:47:52.810441 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:52.804746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.4.28.3:36099)
0805 22:47:52.804977 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:52.804984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:52.805053 (+ 69us) server_negotiation.cc:407] Connection header received
0805 22:47:52.805317 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:52.805322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:52.805424 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:52.805598 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:52.805992 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.806720 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.808096 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:52.808495 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:52.808613 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:52.809132 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:52.809803 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:52.809956 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:52.810043 (+ 87us) server_negotiation.cc:299] Negotiation successful
0805 22:47:52.810134 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 22:47:52.818688 15284 raft_consensus.cc:1261] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Refusing update from remote peer 419750b94684451085a90fe588f31632: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:52.818719 15474 raft_consensus.cc:1261] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Refusing update from remote peer 419750b94684451085a90fe588f31632: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:52.819528 15806 consensus_queue.cc:1035] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:52.819870 15797 consensus_queue.cc:1035] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:52.854609 15060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:52.854948 15060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:52.859546 15060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2: Sending DeleteTablet for 3 replicas of tablet 805a9afab3614df3937a769b3d7600c1
I20260805 22:47:52.860962 15445 tablet_service.cc:1514] Processing DeleteTablet for tablet 805a9afab3614df3937a769b3d7600c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:52 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:58840
I20260805 22:47:52.860989 15652 tablet_service.cc:1514] Processing DeleteTablet for tablet 805a9afab3614df3937a769b3d7600c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:52 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:32916
I20260805 22:47:52.862152 15826 tablet_replica.cc:330] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: stopping tablet replica
I20260805 22:47:52.862221 15257 tablet_service.cc:1514] Processing DeleteTablet for tablet 805a9afab3614df3937a769b3d7600c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:52 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:36870
I20260805 22:47:52.862753 15826 raft_consensus.cc:2229] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:52.863433 15826 raft_consensus.cc:2258] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:52.863837 15062 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:52.864711 15062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:52.866114 15826 ts_tablet_manager.cc:1905] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:52.874224 15826 ts_tablet_manager.cc:1918] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:52.874562 15826 log.cc:1196] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/805a9afab3614df3937a769b3d7600c1
I20260805 22:47:52.875315 15826 ts_tablet_manager.cc:1939] T 805a9afab3614df3937a769b3d7600c1 P 9a7a5aab572240948df5ed5c977d54b9: Deleting consensus metadata
I20260805 22:47:52.875953 15256 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8b2d88026d4e5c81e6f01e3bc84322 (DEFAULT_TABLE table=test-table [id=8a81cd45853740ca83a38cbf5b037ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:52.876390 15651 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8b2d88026d4e5c81e6f01e3bc84322 (DEFAULT_TABLE table=test-table [id=8a81cd45853740ca83a38cbf5b037ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:52.876652 15256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8b2d88026d4e5c81e6f01e3bc84322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:52.876608 15445 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8b2d88026d4e5c81e6f01e3bc84322 (DEFAULT_TABLE table=test-table [id=8a81cd45853740ca83a38cbf5b037ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:52.877122 15651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8b2d88026d4e5c81e6f01e3bc84322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:52.884721 15445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8b2d88026d4e5c81e6f01e3bc84322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:52.885167 15786 tablet_bootstrap.cc:492] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: Bootstrap starting.
I20260805 22:47:52.886549 15043 catalog_manager.cc:4841] TS 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917): tablet 805a9afab3614df3937a769b3d7600c1 (table test-table [id=be084e5696834617a999d4f44ce5acc0]) successfully deleted
I20260805 22:47:52.887151 15786 tablet_bootstrap.cc:654] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:52.892028 15787 tablet_bootstrap.cc:492] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: Bootstrap starting.
I20260805 22:47:52.893885 15788 tablet_bootstrap.cc:492] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: Bootstrap starting.
I20260805 22:47:52.893970 15787 tablet_bootstrap.cc:654] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:52.895921 15788 tablet_bootstrap.cc:654] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:52.899922 15827 tablet_replica.cc:330] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: stopping tablet replica
I20260805 22:47:52.900542 15827 raft_consensus.cc:2229] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:52.902657 15788 tablet_bootstrap.cc:492] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: No bootstrap required, opened a new log
I20260805 22:47:52.902841 15788 ts_tablet_manager.cc:1397] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260805 22:47:52.903467 15828 tablet_replica.cc:330] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: stopping tablet replica
I20260805 22:47:52.903802 15788 raft_consensus.cc:348] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:52.904075 15788 raft_consensus.cc:374] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:52.904037 15828 raft_consensus.cc:2229] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:52.904175 15788 raft_consensus.cc:729] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419750b94684451085a90fe588f31632, State: Initialized, Role: FOLLOWER
I20260805 22:47:52.904462 15788 consensus_queue.cc:260] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:52.904615 15828 raft_consensus.cc:2258] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:52.905757 15788 ts_tablet_manager.cc:1428] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260805 22:47:52.905781 15787 tablet_bootstrap.cc:492] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: No bootstrap required, opened a new log
I20260805 22:47:52.905942 15787 ts_tablet_manager.cc:1397] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260805 22:47:52.906942 15786 tablet_bootstrap.cc:492] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: No bootstrap required, opened a new log
I20260805 22:47:52.907189 15786 ts_tablet_manager.cc:1397] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.008s
I20260805 22:47:52.907408 15827 raft_consensus.cc:2258] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:52.908560 15786 raft_consensus.cc:348] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:52.908987 15786 raft_consensus.cc:374] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:52.909164 15786 raft_consensus.cc:729] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11aedee574e043d1853bb259393b5c22, State: Initialized, Role: FOLLOWER
I20260805 22:47:52.909404 15828 ts_tablet_manager.cc:1905] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:52.909638 15786 consensus_queue.cc:260] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:52.911667 15787 raft_consensus.cc:348] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:52.911999 15787 raft_consensus.cc:374] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:52.912161 15787 raft_consensus.cc:729] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7a5aab572240948df5ed5c977d54b9, State: Initialized, Role: FOLLOWER
I20260805 22:47:52.912540 15787 consensus_queue.cc:260] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:52.912707 15786 ts_tablet_manager.cc:1428] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260805 22:47:52.915742 15787 ts_tablet_manager.cc:1428] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260805 22:47:52.917057 15827 ts_tablet_manager.cc:1905] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:52.919382 15828 ts_tablet_manager.cc:1918] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:52.919576 15828 log.cc:1196] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/805a9afab3614df3937a769b3d7600c1
I20260805 22:47:52.920202 15828 ts_tablet_manager.cc:1939] T 805a9afab3614df3937a769b3d7600c1 P 11aedee574e043d1853bb259393b5c22: Deleting consensus metadata
I20260805 22:47:52.922367 15044 catalog_manager.cc:4841] TS 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761): tablet 805a9afab3614df3937a769b3d7600c1 (table test-table [id=be084e5696834617a999d4f44ce5acc0]) successfully deleted
I20260805 22:47:52.923451 15827 ts_tablet_manager.cc:1918] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:52.923628 15827 log.cc:1196] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/805a9afab3614df3937a769b3d7600c1
I20260805 22:47:52.924119 15827 ts_tablet_manager.cc:1939] T 805a9afab3614df3937a769b3d7600c1 P 419750b94684451085a90fe588f31632: Deleting consensus metadata
I20260805 22:47:52.925751 15044 catalog_manager.cc:4841] TS 419750b94684451085a90fe588f31632 (127.4.28.3:36099): tablet 805a9afab3614df3937a769b3d7600c1 (table test-table [id=be084e5696834617a999d4f44ce5acc0]) successfully deleted
I20260805 22:47:52.957861 15793 raft_consensus.cc:482] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:52.958144 15793 raft_consensus.cc:504] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:52.958972 15793 leader_election.cc:288] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761), 419750b94684451085a90fe588f31632 (127.4.28.3:36099)
I20260805 22:47:52.959734 15284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8b2d88026d4e5c81e6f01e3bc84322" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22" is_pre_election: true
I20260805 22:47:52.959774 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8b2d88026d4e5c81e6f01e3bc84322" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419750b94684451085a90fe588f31632" is_pre_election: true
I20260805 22:47:52.960033 15284 raft_consensus.cc:2454] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7a5aab572240948df5ed5c977d54b9 in term 0.
I20260805 22:47:52.960103 15685 raft_consensus.cc:2454] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7a5aab572240948df5ed5c977d54b9 in term 0.
I20260805 22:47:52.960639 15381 leader_election.cc:302] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419750b94684451085a90fe588f31632, 9a7a5aab572240948df5ed5c977d54b9; no voters:
I20260805 22:47:52.961011 15793 raft_consensus.cc:2781] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:52.961154 15793 raft_consensus.cc:482] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:52.961298 15793 raft_consensus.cc:3037] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:52.963656 15793 raft_consensus.cc:504] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:52.964381 15793 leader_election.cc:288] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 election: Requested vote from peers 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761), 419750b94684451085a90fe588f31632 (127.4.28.3:36099)
I20260805 22:47:52.965142 15284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8b2d88026d4e5c81e6f01e3bc84322" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22"
I20260805 22:47:52.965147 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8b2d88026d4e5c81e6f01e3bc84322" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419750b94684451085a90fe588f31632"
I20260805 22:47:52.965416 15685 raft_consensus.cc:3037] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:52.965409 15284 raft_consensus.cc:3037] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:52.968777 15284 raft_consensus.cc:2454] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7a5aab572240948df5ed5c977d54b9 in term 1.
I20260805 22:47:52.968901 15685 raft_consensus.cc:2454] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7a5aab572240948df5ed5c977d54b9 in term 1.
I20260805 22:47:52.969429 15381 leader_election.cc:302] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aedee574e043d1853bb259393b5c22, 9a7a5aab572240948df5ed5c977d54b9; no voters:
I20260805 22:47:52.969807 15793 raft_consensus.cc:2781] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:52.970361 15793 raft_consensus.cc:686] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 LEADER]: Becoming Leader. State: Replica: 9a7a5aab572240948df5ed5c977d54b9, State: Running, Role: LEADER
I20260805 22:47:52.971019 15793 consensus_queue.cc:237] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:52.975116 15060 catalog_manager.cc:5495] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2). New cstate: current_term: 1 leader_uuid: "9a7a5aab572240948df5ed5c977d54b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } health_report { overall_health: UNKNOWN } } }
Aug 05 22:47:53 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970070, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.62@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:53.043543 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.028437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.4.28.62:37155)
0805 22:47:53.029100 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.029108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.029364 (+ 256us) server_negotiation.cc:407] Connection header received
0805 22:47:53.029562 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.029570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.029627 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.029723 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:53.031111 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.031921 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.033226 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.033724 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.033845 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.037664 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:53.037669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:53.037672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:53.037706 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:53.040491 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:53.041289 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:53.041295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:53.041298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:53.041445 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:53.041888 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:53.041893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:53.041897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:53.042220 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:53.042361 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:53.042850 (+ 489us) server_negotiation.cc:299] Negotiation successful
0805 22:47:53.043190 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":491,"thread_start_us":319,"threads_started":1}
I20260805 22:47:53.043543 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.028406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37155 (local address 127.0.0.1:53906)
0805 22:47:53.029247 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:53.029267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:53.029424 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:53.029782 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:53.029787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:53.030572 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:53.030921 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.030933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.032102 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.032106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.033082 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.033096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.033902 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.033908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.034082 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.034833 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:53.034856 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:53.037432 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:53.040651 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:53.040673 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:53.040683 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:53.041163 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:53.041589 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:53.041594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:53.041598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:53.041768 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:53.042348 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:53.042357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:53.042540 (+ 183us) client_negotiation.cc:769] Sending connection context
0805 22:47:53.042705 (+ 165us) client_negotiation.cc:240] Negotiation successful
0805 22:47:53.043039 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":575,"threads_started":1}
I20260805 22:47:53.050625 15060 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:53906:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260805 22:47:53.052074 15060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:53.060096 15257 tablet_service.cc:1467] Processing CreateTablet for tablet f0a122e4793a4ad494b3eadf3d48c216 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc0033876894805b571a8ef88215495]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 22:47:53.060364 15445 tablet_service.cc:1467] Processing CreateTablet for tablet f0a122e4793a4ad494b3eadf3d48c216 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc0033876894805b571a8ef88215495]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 22:47:53.060781 15257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a122e4793a4ad494b3eadf3d48c216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:53.060910 15445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a122e4793a4ad494b3eadf3d48c216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:53.062132 15652 tablet_service.cc:1467] Processing CreateTablet for tablet f0a122e4793a4ad494b3eadf3d48c216 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc0033876894805b571a8ef88215495]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 22:47:53.062822 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a122e4793a4ad494b3eadf3d48c216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:53.068188 15786 tablet_bootstrap.cc:492] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: Bootstrap starting.
I20260805 22:47:53.068188 15787 tablet_bootstrap.cc:492] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: Bootstrap starting.
I20260805 22:47:53.070003 15787 tablet_bootstrap.cc:654] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:53.070017 15786 tablet_bootstrap.cc:654] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:53.072165 15788 tablet_bootstrap.cc:492] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: Bootstrap starting.
I20260805 22:47:53.074285 15788 tablet_bootstrap.cc:654] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:53.081629 15788 tablet_bootstrap.cc:492] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: No bootstrap required, opened a new log
I20260805 22:47:53.081849 15788 ts_tablet_manager.cc:1397] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260805 22:47:53.082374 15786 tablet_bootstrap.cc:492] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: No bootstrap required, opened a new log
I20260805 22:47:53.082576 15786 ts_tablet_manager.cc:1397] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.001s
I20260805 22:47:53.082775 15788 raft_consensus.cc:348] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.083052 15788 raft_consensus.cc:374] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:53.083158 15788 raft_consensus.cc:729] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419750b94684451085a90fe588f31632, State: Initialized, Role: FOLLOWER
I20260805 22:47:53.083338 15786 raft_consensus.cc:348] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.083441 15788 consensus_queue.cc:260] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.083621 15786 raft_consensus.cc:374] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:53.083705 15786 raft_consensus.cc:729] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11aedee574e043d1853bb259393b5c22, State: Initialized, Role: FOLLOWER
I20260805 22:47:53.083937 15786 consensus_queue.cc:260] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.084120 15788 ts_tablet_manager.cc:1428] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260805 22:47:53.083956 15806 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.084328 15806 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:53.084504 15787 tablet_bootstrap.cc:492] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: No bootstrap required, opened a new log
I20260805 22:47:53.084430 15796 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.084697 15787 ts_tablet_manager.cc:1397] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260805 22:47:53.084794 15796 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:53.086205 15786 ts_tablet_manager.cc:1428] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260805 22:47:53.087169 15787 raft_consensus.cc:348] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.087458 15787 raft_consensus.cc:374] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:53.087553 15787 raft_consensus.cc:729] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7a5aab572240948df5ed5c977d54b9, State: Initialized, Role: FOLLOWER
I20260805 22:47:53.087826 15787 consensus_queue.cc:260] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.088428 15793 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.088814 15793 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:53.088457 15787 ts_tablet_manager.cc:1428] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260805 22:47:53.089550 15796 raft_consensus.cc:482] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:53.089710 15796 raft_consensus.cc:504] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.091678 15796 leader_election.cc:288] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917), 419750b94684451085a90fe588f31632 (127.4.28.3:36099)
I20260805 22:47:53.102077 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.092297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:36099 (local address 127.4.28.1:45109)
0805 22:47:53.094331 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:53.094346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:53.094448 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:53.094945 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:53.094949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:53.094988 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:53.095354 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.095368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.096673 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.096679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.099145 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.099163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.100931 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.100939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.101126 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.101166 (+ 40us) client_negotiation.cc:769] Sending connection context
0805 22:47:53.101335 (+ 169us) client_negotiation.cc:240] Negotiation successful
0805 22:47:53.101575 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1907,"mutex_wait_us":130,"thread_start_us":1464,"threads_started":1}
I20260805 22:47:53.102151 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.092676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:45109 (local address 127.4.28.3:36099)
0805 22:47:53.092837 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.092842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.094447 (+ 1605us) server_negotiation.cc:407] Connection header received
0805 22:47:53.094549 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.094555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.094684 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.094781 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:53.095686 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.096516 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.099344 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.100659 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.100812 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.100960 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:53.101390 (+ 430us) server_negotiation.cc:299] Negotiation successful
0805 22:47:53.101892 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 22:47:53.102530 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.092631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:56385 (local address 127.4.28.2:42917)
0805 22:47:53.092833 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.092840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.094191 (+ 1351us) server_negotiation.cc:407] Connection header received
0805 22:47:53.094591 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.094595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.094688 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.094763 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:47:53.095526 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.096635 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.099755 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.100998 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.101139 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.101244 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:53.101668 (+ 424us) server_negotiation.cc:299] Negotiation successful
0805 22:47:53.101802 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:47:53.102864 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a122e4793a4ad494b3eadf3d48c216" candidate_uuid: "11aedee574e043d1853bb259393b5c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419750b94684451085a90fe588f31632" is_pre_election: true
I20260805 22:47:53.103216 15685 raft_consensus.cc:2454] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11aedee574e043d1853bb259393b5c22 in term 0.
I20260805 22:47:53.103293 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.092190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42917 (local address 127.4.28.1:56385)
0805 22:47:53.094071 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:53.094092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:53.094324 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:53.094809 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:53.094814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:53.094840 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:47:53.095321 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.095339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.096746 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.096751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.099560 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.099579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.101267 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.101274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.101453 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.101487 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 22:47:53.102760 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0805 22:47:53.102909 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1728,"mutex_wait_us":69,"thread_start_us":1887,"threads_started":1}
I20260805 22:47:53.103749 15191 leader_election.cc:302] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aedee574e043d1853bb259393b5c22, 419750b94684451085a90fe588f31632; no voters:
I20260805 22:47:53.104187 15796 raft_consensus.cc:2781] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:53.104118 15474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a122e4793a4ad494b3eadf3d48c216" candidate_uuid: "11aedee574e043d1853bb259393b5c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7a5aab572240948df5ed5c977d54b9" is_pre_election: true
I20260805 22:47:53.104319 15796 raft_consensus.cc:482] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:53.104406 15796 raft_consensus.cc:3037] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:53.104399 15474 raft_consensus.cc:2454] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11aedee574e043d1853bb259393b5c22 in term 0.
I20260805 22:47:53.107789 15796 raft_consensus.cc:504] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.108654 15796 leader_election.cc:288] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [CANDIDATE]: Term 1 election: Requested vote from peers 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917), 419750b94684451085a90fe588f31632 (127.4.28.3:36099)
I20260805 22:47:53.109359 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a122e4793a4ad494b3eadf3d48c216" candidate_uuid: "11aedee574e043d1853bb259393b5c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419750b94684451085a90fe588f31632"
I20260805 22:47:53.109367 15474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a122e4793a4ad494b3eadf3d48c216" candidate_uuid: "11aedee574e043d1853bb259393b5c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7a5aab572240948df5ed5c977d54b9"
I20260805 22:47:53.109651 15474 raft_consensus.cc:3037] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:53.109663 15685 raft_consensus.cc:3037] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:53.111789 15474 raft_consensus.cc:2454] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11aedee574e043d1853bb259393b5c22 in term 1.
I20260805 22:47:53.112233 15190 leader_election.cc:302] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aedee574e043d1853bb259393b5c22, 9a7a5aab572240948df5ed5c977d54b9; no voters:
I20260805 22:47:53.112545 15796 raft_consensus.cc:2781] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:53.112931 15685 raft_consensus.cc:2454] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11aedee574e043d1853bb259393b5c22 in term 1.
I20260805 22:47:53.113055 15796 raft_consensus.cc:686] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [term 1 LEADER]: Becoming Leader. State: Replica: 11aedee574e043d1853bb259393b5c22, State: Running, Role: LEADER
I20260805 22:47:53.113550 15796 consensus_queue.cc:237] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.116202 15851 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: TxnStatusTablet state changed. Reason: New leader 11aedee574e043d1853bb259393b5c22. Latest consensus state: current_term: 1 leader_uuid: "11aedee574e043d1853bb259393b5c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.116657 15851 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: This TxnStatusTablet replica's current role is: LEADER
I20260805 22:47:53.117408 15853 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260805 22:47:53.117694 15853 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260805 22:47:53.117969 15060 catalog_manager.cc:5495] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11aedee574e043d1853bb259393b5c22 (127.4.28.1). New cstate: current_term: 1 leader_uuid: "11aedee574e043d1853bb259393b5c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } health_report { overall_health: UNKNOWN } } }
I20260805 22:47:53.144100 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.135312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.4.28.1:46761)
0805 22:47:53.135509 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.135516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.135891 (+ 375us) server_negotiation.cc:407] Connection header received
0805 22:47:53.136522 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.136528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.136644 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.136796 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:53.137423 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.138670 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.140734 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.141483 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.141664 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.142464 (+ 800us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:53.143366 (+ 902us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:53.143517 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:53.143666 (+ 149us) server_negotiation.cc:299] Negotiation successful
0805 22:47:53.143777 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 22:47:53.144145 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.135339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:46761 (local address 127.0.0.1:36872)
0805 22:47:53.135765 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:53.135786 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:53.136223 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:53.137010 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:53.137016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:53.137031 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:47:53.137233 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.137245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.138849 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.138855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.140532 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.140552 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.141820 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.141827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.142107 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.142200 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:47:53.143492 (+ 1292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:47:53.143526 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 22:47:53.143626 (+ 100us) client_negotiation.cc:240] Negotiation successful
0805 22:47:53.143811 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253}
I20260805 22:47:53.149587 15474 raft_consensus.cc:1261] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Refusing update from remote peer 11aedee574e043d1853bb259393b5c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:53.149814 15685 raft_consensus.cc:1261] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Refusing update from remote peer 11aedee574e043d1853bb259393b5c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:53.150774 15796 consensus_queue.cc:1035] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:53.151700 15796 consensus_queue.cc:1035] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:53.158802 15793 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: TxnStatusTablet state changed. Reason: New leader 11aedee574e043d1853bb259393b5c22. Latest consensus state: current_term: 1 leader_uuid: "11aedee574e043d1853bb259393b5c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.159183 15793 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:53.161805 15796 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11aedee574e043d1853bb259393b5c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.162190 15796 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: This TxnStatusTablet replica's current role is: LEADER
I20260805 22:47:53.163735 15811 mvcc.cc:205] Tried to move back new op lower bound from 7315333419610992640 to 7315333419478339584. Current Snapshot: MvccSnapshot[applied={T|T < 7315333419610992640 or (T in {7315333419610992640})}]
I20260805 22:47:53.164522 15793 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11aedee574e043d1853bb259393b5c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.164743 15813 mvcc.cc:205] Tried to move back new op lower bound from 7315333419610992640 to 7315333419478339584. Current Snapshot: MvccSnapshot[applied={T|T < 7315333419610992640}]
I20260805 22:47:53.164835 15793 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 9a7a5aab572240948df5ed5c977d54b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:53.175707 15806 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: TxnStatusTablet state changed. Reason: New leader 11aedee574e043d1853bb259393b5c22. Latest consensus state: current_term: 1 leader_uuid: "11aedee574e043d1853bb259393b5c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.176163 15806 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:53.179324 15796 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11aedee574e043d1853bb259393b5c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.179630 15796 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 11aedee574e043d1853bb259393b5c22: This TxnStatusTablet replica's current role is: LEADER
I20260805 22:47:53.185288 15806 tablet_replica.cc:439] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11aedee574e043d1853bb259393b5c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } } }
I20260805 22:47:53.185611 15806 tablet_replica.cc:442] T f0a122e4793a4ad494b3eadf3d48c216 P 419750b94684451085a90fe588f31632: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 22:47:53.190127 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.179298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.4.28.2:42917)
0805 22:47:53.179501 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.179507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.179526 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:47:53.179649 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.179654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.179769 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.179935 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:53.180828 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.182253 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.184421 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.185117 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.185246 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.187915 (+ 2669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:53.188713 (+ 798us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:53.188924 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:53.189680 (+ 756us) server_negotiation.cc:299] Negotiation successful
0805 22:47:53.189780 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 22:47:53.223392 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.216092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:59183 (local address 127.4.28.1:46761)
0805 22:47:53.216318 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.216324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.218261 (+ 1937us) server_negotiation.cc:407] Connection header received
0805 22:47:53.218481 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.218486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.218583 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.218662 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:53.219225 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.220037 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.221493 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.222014 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.222136 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.222633 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:53.222754 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:53.222832 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:53.222986 (+ 154us) server_negotiation.cc:299] Negotiation successful
0805 22:47:53.223074 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260805 22:47:53.223755 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.216106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:46761 (local address 127.4.28.2:59183)
0805 22:47:53.218133 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:53.218152 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:53.218327 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:53.218711 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:53.218716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:53.218729 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:47:53.219048 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.219058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.220190 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.220195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.221335 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.221349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.222217 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.222223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.222359 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.222428 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:47:53.222859 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:47:53.222872 (+ 13us) client_negotiation.cc:769] Sending connection context
0805 22:47:53.222945 (+ 73us) client_negotiation.cc:240] Negotiation successful
0805 22:47:53.223109 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1836,"thread_start_us":1672,"threads_started":1}
I20260805 22:47:53.236377 15284 raft_consensus.cc:1261] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Refusing update from remote peer 9a7a5aab572240948df5ed5c977d54b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:53.236878 15685 raft_consensus.cc:1261] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Refusing update from remote peer 9a7a5aab572240948df5ed5c977d54b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:53.237859 15833 consensus_queue.cc:1035] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:53.238236 15793 consensus_queue.cc:1035] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:53.276973 15810 mvcc.cc:205] Tried to move back new op lower bound from 7315333419967610880 to 7315333418893529088. Current Snapshot: MvccSnapshot[applied={T|T < 7315333419967610880}]
I20260805 22:47:53.324000 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.307436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:43669 (local address 127.4.28.2:42917)
0805 22:47:53.307643 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.307649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.314276 (+ 6627us) server_negotiation.cc:407] Connection header received
0805 22:47:53.314526 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.314532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.314627 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.314706 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:53.315470 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.316549 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.321535 (+ 4986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.322259 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.322383 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.323150 (+ 767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:53.323319 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:53.323382 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:53.323597 (+ 215us) server_negotiation.cc:299] Negotiation successful
0805 22:47:53.323695 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260805 22:47:53.325150 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.307062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:42917 (local address 127.4.28.1:43669)
0805 22:47:53.314126 (+ 7064us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:53.314152 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:53.314333 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:53.314882 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:53.314887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:53.314902 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 22:47:53.315250 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.315262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.316750 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.316756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.321300 (+ 4544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:53.321322 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.322579 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.322585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:53.322793 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.322901 (+ 108us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 22:47:53.323470 (+ 569us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 22:47:53.323480 (+ 10us) client_negotiation.cc:769] Sending connection context
0805 22:47:53.324515 (+ 1035us) client_negotiation.cc:240] Negotiation successful
0805 22:47:53.324706 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6803,"thread_start_us":6613,"threads_started":1}
I20260805 22:47:53.400806 15060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:53.401145 15060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:53.405351 15060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2: Sending DeleteTablet for 3 replicas of tablet 8d8b2d88026d4e5c81e6f01e3bc84322
I20260805 22:47:53.406514 15445 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8b2d88026d4e5c81e6f01e3bc84322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:53 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:58840
I20260805 22:47:53.407032 15257 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8b2d88026d4e5c81e6f01e3bc84322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:53 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:36870
I20260805 22:47:53.407371 15828 tablet_replica.cc:330] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: stopping tablet replica
I20260805 22:47:53.407612 15828 raft_consensus.cc:2229] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:53.407692 15652 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8b2d88026d4e5c81e6f01e3bc84322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:53 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:32916
I20260805 22:47:53.407827 15828 raft_consensus.cc:2258] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:53.408017 15827 tablet_replica.cc:330] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: stopping tablet replica
I20260805 22:47:53.408252 15827 raft_consensus.cc:2229] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:53.408507 15827 raft_consensus.cc:2258] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:53.409271 15828 ts_tablet_manager.cc:1905] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:53.410128 15827 ts_tablet_manager.cc:1905] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:53.420773 15827 ts_tablet_manager.cc:1918] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:53.420982 15827 log.cc:1196] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/8d8b2d88026d4e5c81e6f01e3bc84322
I20260805 22:47:53.421398 15827 ts_tablet_manager.cc:1939] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 419750b94684451085a90fe588f31632: Deleting consensus metadata
I20260805 22:47:53.428435 15044 catalog_manager.cc:4841] TS 419750b94684451085a90fe588f31632 (127.4.28.3:36099): tablet 8d8b2d88026d4e5c81e6f01e3bc84322 (table test-table [id=8a81cd45853740ca83a38cbf5b037ee0]) successfully deleted
I20260805 22:47:53.431442 15871 tablet_replica.cc:330] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: stopping tablet replica
I20260805 22:47:53.431697 15871 raft_consensus.cc:2229] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:53.432293 15871 raft_consensus.cc:2258] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:53.432868 15828 ts_tablet_manager.cc:1918] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:53.433149 15828 log.cc:1196] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/8d8b2d88026d4e5c81e6f01e3bc84322
I20260805 22:47:53.433753 15828 ts_tablet_manager.cc:1939] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 11aedee574e043d1853bb259393b5c22: Deleting consensus metadata
I20260805 22:47:53.433781 15871 ts_tablet_manager.cc:1905] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:53.435194 15044 catalog_manager.cc:4841] TS 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761): tablet 8d8b2d88026d4e5c81e6f01e3bc84322 (table test-table [id=8a81cd45853740ca83a38cbf5b037ee0]) successfully deleted
I20260805 22:47:53.441663 15871 ts_tablet_manager.cc:1918] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:53.441838 15871 log.cc:1196] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/8d8b2d88026d4e5c81e6f01e3bc84322
I20260805 22:47:53.442327 15871 ts_tablet_manager.cc:1939] T 8d8b2d88026d4e5c81e6f01e3bc84322 P 9a7a5aab572240948df5ed5c977d54b9: Deleting consensus metadata
I20260805 22:47:53.443702 15043 catalog_manager.cc:4841] TS 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917): tablet 8d8b2d88026d4e5c81e6f01e3bc84322 (table test-table [id=8a81cd45853740ca83a38cbf5b037ee0]) successfully deleted
Aug 05 22:47:53 dist-test-slave-7lbf krb5kdc[15003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 22:47:53 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:47:53 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
W20260805 22:47:53.496304 15842 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:47:53.485084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.4.28.62:37155)
0805 22:47:53.485323 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.485330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.485903 (+ 573us) server_negotiation.cc:407] Connection header received
0805 22:47:53.486100 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.486106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.486179 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.486267 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:53.487884 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.488643 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.490255 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.490789 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.490917 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.496026 (+ 5109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53920: BlockingRecv error: recv got EOF from 127.0.0.1:53920 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 05 22:47:53 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM
I20260805 22:47:53.541821 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.526400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.4.28.62:37155)
0805 22:47:53.526650 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.526658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.527325 (+ 667us) server_negotiation.cc:407] Connection header received
0805 22:47:53.527530 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.527537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.527614 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.527701 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:53.529087 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.529944 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.531288 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.531726 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.531848 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.536061 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:53.536068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:53.536071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:53.536115 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:53.538872 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:53.539634 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:53.539638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:53.539641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:53.539792 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:53.540207 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:53.540212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:53.540214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:53.540532 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:53.540790 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:53.541123 (+ 333us) server_negotiation.cc:299] Negotiation successful
0805 22:47:53.541400 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260805 22:47:53.545778 15060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:53.546483 15060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:53.554445 15257 tablet_service.cc:1467] Processing CreateTablet for tablet 90a92cfb2f734c6296f23feaf4265973 (DEFAULT_TABLE table=test-table [id=4bdb7260922749698a4b5647547df458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:53.554682 15445 tablet_service.cc:1467] Processing CreateTablet for tablet 90a92cfb2f734c6296f23feaf4265973 (DEFAULT_TABLE table=test-table [id=4bdb7260922749698a4b5647547df458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:53.554996 15257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a92cfb2f734c6296f23feaf4265973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:53.555109 15445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a92cfb2f734c6296f23feaf4265973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:53.558604 15652 tablet_service.cc:1467] Processing CreateTablet for tablet 90a92cfb2f734c6296f23feaf4265973 (DEFAULT_TABLE table=test-table [id=4bdb7260922749698a4b5647547df458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:53.559147 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a92cfb2f734c6296f23feaf4265973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:53.564545 15787 tablet_bootstrap.cc:492] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: Bootstrap starting.
I20260805 22:47:53.566484 15787 tablet_bootstrap.cc:654] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:53.570734 15786 tablet_bootstrap.cc:492] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: Bootstrap starting.
I20260805 22:47:53.571331 15788 tablet_bootstrap.cc:492] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: Bootstrap starting.
I20260805 22:47:53.572731 15786 tablet_bootstrap.cc:654] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:53.572999 15788 tablet_bootstrap.cc:654] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:53.583709 15788 tablet_bootstrap.cc:492] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: No bootstrap required, opened a new log
I20260805 22:47:53.583918 15788 ts_tablet_manager.cc:1397] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260805 22:47:53.584744 15787 tablet_bootstrap.cc:492] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: No bootstrap required, opened a new log
I20260805 22:47:53.584798 15788 raft_consensus.cc:348] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.585086 15788 raft_consensus.cc:374] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:53.585196 15788 raft_consensus.cc:729] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419750b94684451085a90fe588f31632, State: Initialized, Role: FOLLOWER
I20260805 22:47:53.585558 15787 ts_tablet_manager.cc:1397] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.004s
I20260805 22:47:53.585446 15788 consensus_queue.cc:260] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.586102 15788 ts_tablet_manager.cc:1428] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260805 22:47:53.588675 15787 raft_consensus.cc:348] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.589869 15787 raft_consensus.cc:374] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:53.590260 15787 raft_consensus.cc:729] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7a5aab572240948df5ed5c977d54b9, State: Initialized, Role: FOLLOWER
I20260805 22:47:53.590940 15787 consensus_queue.cc:260] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.594908 15787 ts_tablet_manager.cc:1428] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260805 22:47:53.596720 15786 tablet_bootstrap.cc:492] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: No bootstrap required, opened a new log
I20260805 22:47:53.596941 15786 ts_tablet_manager.cc:1397] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20260805 22:47:53.597915 15786 raft_consensus.cc:348] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.598251 15786 raft_consensus.cc:374] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:53.598369 15786 raft_consensus.cc:729] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11aedee574e043d1853bb259393b5c22, State: Initialized, Role: FOLLOWER
I20260805 22:47:53.598666 15786 consensus_queue.cc:260] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.599303 15786 ts_tablet_manager.cc:1428] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260805 22:47:53.802518 15806 raft_consensus.cc:482] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:53.802837 15806 raft_consensus.cc:504] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.803607 15806 leader_election.cc:288] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761), 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917)
I20260805 22:47:53.804301 15471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a92cfb2f734c6296f23feaf4265973" candidate_uuid: "419750b94684451085a90fe588f31632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7a5aab572240948df5ed5c977d54b9" is_pre_election: true
I20260805 22:47:53.804337 15283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a92cfb2f734c6296f23feaf4265973" candidate_uuid: "419750b94684451085a90fe588f31632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22" is_pre_election: true
I20260805 22:47:53.804639 15471 raft_consensus.cc:2454] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 419750b94684451085a90fe588f31632 in term 0.
I20260805 22:47:53.804648 15283 raft_consensus.cc:2454] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 419750b94684451085a90fe588f31632 in term 0.
I20260805 22:47:53.805171 15590 leader_election.cc:302] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419750b94684451085a90fe588f31632, 9a7a5aab572240948df5ed5c977d54b9; no voters:
I20260805 22:47:53.805482 15806 raft_consensus.cc:2781] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:53.805608 15806 raft_consensus.cc:482] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:53.805717 15806 raft_consensus.cc:3037] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:53.808050 15806 raft_consensus.cc:504] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.808653 15806 leader_election.cc:288] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [CANDIDATE]: Term 1 election: Requested vote from peers 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761), 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917)
I20260805 22:47:53.809289 15471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a92cfb2f734c6296f23feaf4265973" candidate_uuid: "419750b94684451085a90fe588f31632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7a5aab572240948df5ed5c977d54b9"
I20260805 22:47:53.809298 15283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a92cfb2f734c6296f23feaf4265973" candidate_uuid: "419750b94684451085a90fe588f31632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22"
I20260805 22:47:53.809506 15471 raft_consensus.cc:3037] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:53.809554 15283 raft_consensus.cc:3037] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:53.811899 15471 raft_consensus.cc:2454] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 419750b94684451085a90fe588f31632 in term 1.
I20260805 22:47:53.812391 15590 leader_election.cc:302] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419750b94684451085a90fe588f31632, 9a7a5aab572240948df5ed5c977d54b9; no voters:
I20260805 22:47:53.812665 15806 raft_consensus.cc:2781] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:53.812629 15283 raft_consensus.cc:2454] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 419750b94684451085a90fe588f31632 in term 1.
I20260805 22:47:53.815404 15806 raft_consensus.cc:686] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 1 LEADER]: Becoming Leader. State: Replica: 419750b94684451085a90fe588f31632, State: Running, Role: LEADER
I20260805 22:47:53.815914 15806 consensus_queue.cc:237] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.819293 15062 catalog_manager.cc:5495] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 reported cstate change: term changed from 0 to 1, leader changed from <none> to 419750b94684451085a90fe588f31632 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "419750b94684451085a90fe588f31632" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } health_report { overall_health: HEALTHY } } }
I20260805 22:47:53.844772 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:53.838817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.4.28.3:36099)
0805 22:47:53.839441 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:53.839447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:53.839464 (+ 17us) server_negotiation.cc:407] Connection header received
0805 22:47:53.839525 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:53.839528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:53.839592 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:53.839680 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:53.840099 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.840893 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.842479 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:53.843235 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:53.843388 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:53.843848 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:53.844015 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:53.844120 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:53.844271 (+ 151us) server_negotiation.cc:299] Negotiation successful
0805 22:47:53.844368 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":356,"threads_started":1}
I20260805 22:47:53.848193 15283 raft_consensus.cc:1261] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Refusing update from remote peer 419750b94684451085a90fe588f31632: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:53.848273 15471 raft_consensus.cc:1261] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Refusing update from remote peer 419750b94684451085a90fe588f31632: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:53.848953 15806 consensus_queue.cc:1035] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:53.849236 15894 consensus_queue.cc:1035] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:53.865092 15060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:53.865365 15060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:53.869228 15060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2: Sending DeleteTablet for 3 replicas of tablet 90a92cfb2f734c6296f23feaf4265973
I20260805 22:47:53.870335 15445 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a92cfb2f734c6296f23feaf4265973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:53 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:58840
I20260805 22:47:53.870604 15652 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a92cfb2f734c6296f23feaf4265973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:53 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:32916
I20260805 22:47:53.870724 15257 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a92cfb2f734c6296f23feaf4265973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:53 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:36870
I20260805 22:47:53.870970 15871 tablet_replica.cc:330] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: stopping tablet replica
I20260805 22:47:53.871054 15828 tablet_replica.cc:330] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: stopping tablet replica
I20260805 22:47:53.871186 15871 raft_consensus.cc:2229] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:53.871239 15828 raft_consensus.cc:2229] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:53.871459 15871 raft_consensus.cc:2258] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:53.871488 15828 raft_consensus.cc:2258] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:53.871809 15827 tablet_replica.cc:330] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: stopping tablet replica
I20260805 22:47:53.872006 15827 raft_consensus.cc:2229] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:53.872332 15827 raft_consensus.cc:2258] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:53.872150 15060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:47:53.872908 15060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:47:53.873659 15827 ts_tablet_manager.cc:1905] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:53.873996 15871 ts_tablet_manager.cc:1905] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:53.874464 15828 ts_tablet_manager.cc:1905] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:53.880590 15827 ts_tablet_manager.cc:1918] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:53.880757 15827 log.cc:1196] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/90a92cfb2f734c6296f23feaf4265973
I20260805 22:47:53.881186 15827 ts_tablet_manager.cc:1939] T 90a92cfb2f734c6296f23feaf4265973 P 419750b94684451085a90fe588f31632: Deleting consensus metadata
I20260805 22:47:53.881268 15828 ts_tablet_manager.cc:1918] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:53.881429 15828 log.cc:1196] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/90a92cfb2f734c6296f23feaf4265973
I20260805 22:47:53.882486 15871 ts_tablet_manager.cc:1918] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:47:53.882539 15044 catalog_manager.cc:4841] TS 419750b94684451085a90fe588f31632 (127.4.28.3:36099): tablet 90a92cfb2f734c6296f23feaf4265973 (table test-table [id=4bdb7260922749698a4b5647547df458]) successfully deleted
I20260805 22:47:53.882774 15871 log.cc:1196] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/90a92cfb2f734c6296f23feaf4265973
I20260805 22:47:53.883250 15871 ts_tablet_manager.cc:1939] T 90a92cfb2f734c6296f23feaf4265973 P 9a7a5aab572240948df5ed5c977d54b9: Deleting consensus metadata
I20260805 22:47:53.884450 15043 catalog_manager.cc:4841] TS 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917): tablet 90a92cfb2f734c6296f23feaf4265973 (table test-table [id=4bdb7260922749698a4b5647547df458]) successfully deleted
I20260805 22:47:53.885434 15828 ts_tablet_manager.cc:1939] T 90a92cfb2f734c6296f23feaf4265973 P 11aedee574e043d1853bb259393b5c22: Deleting consensus metadata
I20260805 22:47:53.886718 15044 catalog_manager.cc:4841] TS 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761): tablet 90a92cfb2f734c6296f23feaf4265973 (table test-table [id=4bdb7260922749698a4b5647547df458]) successfully deleted
I20260805 22:47:53.889608 15257 tablet_service.cc:1467] Processing CreateTablet for tablet a47879c207104a679fe0feab2f9d82ba (DEFAULT_TABLE table=test-table [id=f1e194a2f0484a898acbea9b779d6fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:53.890133 15257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47879c207104a679fe0feab2f9d82ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:53.893994 15445 tablet_service.cc:1467] Processing CreateTablet for tablet a47879c207104a679fe0feab2f9d82ba (DEFAULT_TABLE table=test-table [id=f1e194a2f0484a898acbea9b779d6fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:53.894488 15445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47879c207104a679fe0feab2f9d82ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:53.897049 15786 tablet_bootstrap.cc:492] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: Bootstrap starting.
I20260805 22:47:53.898824 15786 tablet_bootstrap.cc:654] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:53.901698 15787 tablet_bootstrap.cc:492] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: Bootstrap starting.
I20260805 22:47:53.902607 15652 tablet_service.cc:1467] Processing CreateTablet for tablet a47879c207104a679fe0feab2f9d82ba (DEFAULT_TABLE table=test-table [id=f1e194a2f0484a898acbea9b779d6fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:47:53.903064 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47879c207104a679fe0feab2f9d82ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:47:53.903697 15787 tablet_bootstrap.cc:654] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:53.906180 15787 tablet_bootstrap.cc:492] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: No bootstrap required, opened a new log
I20260805 22:47:53.906381 15787 ts_tablet_manager.cc:1397] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260805 22:47:53.906502 15786 tablet_bootstrap.cc:492] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: No bootstrap required, opened a new log
I20260805 22:47:53.906702 15786 ts_tablet_manager.cc:1397] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260805 22:47:53.907277 15787 raft_consensus.cc:348] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.907567 15787 raft_consensus.cc:374] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:53.907655 15787 raft_consensus.cc:729] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7a5aab572240948df5ed5c977d54b9, State: Initialized, Role: FOLLOWER
I20260805 22:47:53.907486 15786 raft_consensus.cc:348] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.907897 15786 raft_consensus.cc:374] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:53.908036 15786 raft_consensus.cc:729] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11aedee574e043d1853bb259393b5c22, State: Initialized, Role: FOLLOWER
I20260805 22:47:53.907935 15787 consensus_queue.cc:260] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.908344 15786 consensus_queue.cc:260] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.909267 15787 ts_tablet_manager.cc:1428] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260805 22:47:53.909833 15786 ts_tablet_manager.cc:1428] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260805 22:47:53.911392 15788 tablet_bootstrap.cc:492] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: Bootstrap starting.
I20260805 22:47:53.913286 15788 tablet_bootstrap.cc:654] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:53.919018 15788 tablet_bootstrap.cc:492] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: No bootstrap required, opened a new log
I20260805 22:47:53.919238 15788 ts_tablet_manager.cc:1397] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260805 22:47:53.920045 15788 raft_consensus.cc:348] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.920336 15788 raft_consensus.cc:374] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:53.920418 15788 raft_consensus.cc:729] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419750b94684451085a90fe588f31632, State: Initialized, Role: FOLLOWER
I20260805 22:47:53.920670 15788 consensus_queue.cc:260] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:53.921228 15788 ts_tablet_manager.cc:1428] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260805 22:47:54.019649 15833 raft_consensus.cc:482] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:47:54.020004 15833 raft_consensus.cc:504] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:54.020839 15833 leader_election.cc:288] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761), 419750b94684451085a90fe588f31632 (127.4.28.3:36099)
I20260805 22:47:54.021704 15283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47879c207104a679fe0feab2f9d82ba" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22" is_pre_election: true
I20260805 22:47:54.021703 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47879c207104a679fe0feab2f9d82ba" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419750b94684451085a90fe588f31632" is_pre_election: true
I20260805 22:47:54.021992 15685 raft_consensus.cc:2454] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7a5aab572240948df5ed5c977d54b9 in term 0.
I20260805 22:47:54.022022 15283 raft_consensus.cc:2454] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7a5aab572240948df5ed5c977d54b9 in term 0.
I20260805 22:47:54.022521 15381 leader_election.cc:302] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aedee574e043d1853bb259393b5c22, 9a7a5aab572240948df5ed5c977d54b9; no voters:
I20260805 22:47:54.022853 15833 raft_consensus.cc:2781] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:47:54.022984 15833 raft_consensus.cc:482] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:47:54.023097 15833 raft_consensus.cc:3037] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:54.025213 15833 raft_consensus.cc:504] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:54.025874 15833 leader_election.cc:288] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 election: Requested vote from peers 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761), 419750b94684451085a90fe588f31632 (127.4.28.3:36099)
I20260805 22:47:54.026500 15283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47879c207104a679fe0feab2f9d82ba" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aedee574e043d1853bb259393b5c22"
I20260805 22:47:54.026618 15685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47879c207104a679fe0feab2f9d82ba" candidate_uuid: "9a7a5aab572240948df5ed5c977d54b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419750b94684451085a90fe588f31632"
I20260805 22:47:54.026746 15283 raft_consensus.cc:3037] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:54.026873 15685 raft_consensus.cc:3037] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:54.029922 15283 raft_consensus.cc:2454] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7a5aab572240948df5ed5c977d54b9 in term 1.
I20260805 22:47:54.029912 15685 raft_consensus.cc:2454] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7a5aab572240948df5ed5c977d54b9 in term 1.
I20260805 22:47:54.030438 15381 leader_election.cc:302] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aedee574e043d1853bb259393b5c22, 9a7a5aab572240948df5ed5c977d54b9; no voters:
I20260805 22:47:54.030810 15833 raft_consensus.cc:2781] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:54.031250 15833 raft_consensus.cc:686] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 1 LEADER]: Becoming Leader. State: Replica: 9a7a5aab572240948df5ed5c977d54b9, State: Running, Role: LEADER
I20260805 22:47:54.031543 15833 consensus_queue.cc:237] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } }
I20260805 22:47:54.034253 15059 catalog_manager.cc:5495] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2). New cstate: current_term: 1 leader_uuid: "9a7a5aab572240948df5ed5c977d54b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a7a5aab572240948df5ed5c977d54b9" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 42917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 } health_report { overall_health: UNKNOWN } } }
I20260805 22:47:54.080758 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:54.075508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.4.28.2:42917)
0805 22:47:54.076068 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:54.076073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:54.076089 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:47:54.076146 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:54.076149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:54.076215 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:54.076302 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:47:54.076725 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:54.077557 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:54.078937 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:54.079384 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:54.079513 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:54.079978 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:47:54.080100 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:47:54.080181 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:54.080332 (+ 151us) server_negotiation.cc:299] Negotiation successful
0805 22:47:54.080422 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":305,"threads_started":1}
I20260805 22:47:54.090646 15685 raft_consensus.cc:1261] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Refusing update from remote peer 9a7a5aab572240948df5ed5c977d54b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:54.090718 15283 raft_consensus.cc:1261] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Refusing update from remote peer 9a7a5aab572240948df5ed5c977d54b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:47:54.091216 15907 consensus_queue.cc:1035] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419750b94684451085a90fe588f31632" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 36099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:54.091565 15833 consensus_queue.cc:1035] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11aedee574e043d1853bb259393b5c22" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:47:54.146047 15059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:54.146348 15059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:47:54.149866 15059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a67a65f70e9426b9720b38dfd8a21e2: Sending DeleteTablet for 3 replicas of tablet a47879c207104a679fe0feab2f9d82ba
I20260805 22:47:54.150846 15257 tablet_service.cc:1514] Processing DeleteTablet for tablet a47879c207104a679fe0feab2f9d82ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:54 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:36870
I20260805 22:47:54.150856 15652 tablet_service.cc:1514] Processing DeleteTablet for tablet a47879c207104a679fe0feab2f9d82ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:54 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:32916
I20260805 22:47:54.150897 15445 tablet_service.cc:1514] Processing DeleteTablet for tablet a47879c207104a679fe0feab2f9d82ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:47:54 UTC) from {username='oryx', principal='oryx/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:58840
I20260805 22:47:54.151192 15828 tablet_replica.cc:330] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: stopping tablet replica
I20260805 22:47:54.151230 15871 tablet_replica.cc:330] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: stopping tablet replica
I20260805 22:47:54.151278 15827 tablet_replica.cc:330] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: stopping tablet replica
I20260805 22:47:54.151419 15828 raft_consensus.cc:2229] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:54.151422 15871 raft_consensus.cc:2229] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:47:54.151476 15827 raft_consensus.cc:2229] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:47:54.151629 15828 raft_consensus.cc:2258] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:54.151684 15827 raft_consensus.cc:2258] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:54.151780 15871 raft_consensus.cc:2258] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:47:54.152911 15871 ts_tablet_manager.cc:1905] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:54.152930 15828 ts_tablet_manager.cc:1905] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:54.153333 15827 ts_tablet_manager.cc:1905] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:47:54.155680 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 15165
I20260805 22:47:54.159955 15828 ts_tablet_manager.cc:1918] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:54.160161 15828 log.cc:1196] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/a47879c207104a679fe0feab2f9d82ba
I20260805 22:47:54.160630 15828 ts_tablet_manager.cc:1939] T a47879c207104a679fe0feab2f9d82ba P 11aedee574e043d1853bb259393b5c22: Deleting consensus metadata
I20260805 22:47:54.160615 15827 ts_tablet_manager.cc:1918] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:54.160894 15827 log.cc:1196] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/a47879c207104a679fe0feab2f9d82ba
I20260805 22:47:54.161423 15827 ts_tablet_manager.cc:1939] T a47879c207104a679fe0feab2f9d82ba P 419750b94684451085a90fe588f31632: Deleting consensus metadata
I20260805 22:47:54.161867 15871 ts_tablet_manager.cc:1918] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 22:47:54.162055 15871 log.cc:1196] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/a47879c207104a679fe0feab2f9d82ba
I20260805 22:47:54.162484 15871 ts_tablet_manager.cc:1939] T a47879c207104a679fe0feab2f9d82ba P 9a7a5aab572240948df5ed5c977d54b9: Deleting consensus metadata
I20260805 22:47:54.163139 15044 catalog_manager.cc:4841] TS 419750b94684451085a90fe588f31632 (127.4.28.3:36099): tablet a47879c207104a679fe0feab2f9d82ba (table test-table [id=f1e194a2f0484a898acbea9b779d6fb3]) successfully deleted
I20260805 22:47:54.163816 15043 catalog_manager.cc:4841] TS 9a7a5aab572240948df5ed5c977d54b9 (127.4.28.2:42917): tablet a47879c207104a679fe0feab2f9d82ba (table test-table [id=f1e194a2f0484a898acbea9b779d6fb3]) successfully deleted
I20260805 22:47:54.166070 15044 catalog_manager.cc:4841] TS 11aedee574e043d1853bb259393b5c22 (127.4.28.1:46761): tablet a47879c207104a679fe0feab2f9d82ba (table test-table [id=f1e194a2f0484a898acbea9b779d6fb3]) successfully deleted
Aug 05 22:47:54 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.1@KRBTEST.COM
I20260805 22:47:54.183025 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:54.156983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.4.28.1:46761)
0805 22:47:54.166142 (+ 9159us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:54.166149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:54.166842 (+ 693us) server_negotiation.cc:407] Connection header received
0805 22:47:54.167030 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:54.167034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:54.167112 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:54.167215 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:54.168547 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:54.169368 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:54.170764 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:54.171230 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:54.171352 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:54.175983 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:54.176002 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:54.176023 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:54.176074 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:54.179585 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:54.180507 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:54.180517 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:54.180520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:54.180709 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:54.181171 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:54.181176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:54.181178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:54.181657 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:54.182025 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:54.182352 (+ 327us) server_negotiation.cc:299] Negotiation successful
0805 22:47:54.182603 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8903,"thread_start_us":8695,"threads_started":1}
I20260805 22:47:54.183786 15345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:54.311944 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 15165
I20260805 22:47:54.348253 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 15359
Aug 05 22:47:54 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.2@KRBTEST.COM
I20260805 22:47:54.364928 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:54.349088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.4.28.2:42917)
0805 22:47:54.349266 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:54.349271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:54.349290 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:47:54.349422 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:54.349427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:54.349509 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:54.349622 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:54.350982 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:54.351750 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:54.353045 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:54.353455 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:54.353737 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:54.358341 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:54.358375 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:54.358397 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:54.358442 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:54.361370 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:54.362368 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:54.362378 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:54.362380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:54.362567 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:54.363134 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:54.363139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:54.363142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:54.363600 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:54.363879 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:54.364279 (+ 400us) server_negotiation.cc:299] Negotiation successful
0805 22:47:54.364486 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:47:54.365722 15550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:54.492128 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 15359
I20260805 22:47:54.528566 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 15563
Aug 05 22:47:54 dist-test-slave-7lbf krb5kdc[15003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.28.3@KRBTEST.COM
I20260805 22:47:54.545112 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:54.529429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.4.28.3:36099)
0805 22:47:54.529957 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:54.529962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:54.529980 (+ 18us) server_negotiation.cc:407] Connection header received
0805 22:47:54.530041 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:54.530045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:54.530108 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:54.530202 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:54.531535 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:54.532257 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:54.533711 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:54.534142 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:54.534257 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:54.538595 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:54.538613 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:54.538632 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:54.538678 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:54.541816 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:54.542644 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:54.542672 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:54.542676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:54.542848 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:54.543339 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:54.543343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:54.543345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:54.543751 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:54.544001 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:54.544472 (+ 471us) server_negotiation.cc:299] Negotiation successful
0805 22:47:54.544728 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":273,"threads_started":1}
I20260805 22:47:54.545912 15742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:54.670295 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 15563
I20260805 22:47:54.698683 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 15019
I20260805 22:47:54.700057 15115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:47:54.842808 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 15019
2026-08-05T22:47:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6287 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:54 dist-test-slave-7lbf krb5kdc[15938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:54 dist-test-slave-7lbf krb5kdc[15938](info): set up 2 sockets
Aug 05 22:47:54 dist-test-slave-7lbf krb5kdc[15938](info): commencing operation
krb5kdc: starting...
W20260805 22:47:56.046984 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:56 dist-test-slave-7lbf krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:47:56.338585 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:39535
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:39535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:56.693776 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:56.694123 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:56.694259 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:56.701756 15956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:56.701889 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:56.701967 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:56.702046 15956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:56.702116 15956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:56.712491 15956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:39535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:39535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:56.714005 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:56.715438 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:56.727919 15966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:56.728283 15967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:56.728312 15969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:56.729351 15956 server_base.cc:1034] running on GCE node
I20260805 22:47:56.730333 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:56.731953 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:56.733222 15956 hybrid_clock.cc:648] HybridClock initialized: now 1785970076733156 us; error 50 us; skew 500 ppm
I20260805 22:47:56.733656 15956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-05T22:47:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1967 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:47:56 dist-test-slave-7lbf krb5kdc[15978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:47:56 dist-test-slave-7lbf krb5kdc[15978](info): set up 2 sockets
Aug 05 22:47:56 dist-test-slave-7lbf krb5kdc[15978](info): commencing operation
krb5kdc: starting...
W20260805 22:47:58.026629 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:47:58 dist-test-slave-7lbf krb5kdc[15978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:47:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:47:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:47:58.314191 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44295
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:44295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:58.666841 15996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:58.667189 15996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:58.667315 15996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:58.675247 15996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:47:58.675395 15996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:58.675503 15996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:58.675593 15996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:47:58.675657 15996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:47:58.686224 15996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:44295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:58.687695 15996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:58.689085 15996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:58.703274 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:58.703227 16002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:58.703253 16006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:58.707244 15996 server_base.cc:1034] running on GCE node
I20260805 22:47:58.708192 15996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:58.709913 15996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:58.711161 15996 hybrid_clock.cc:648] HybridClock initialized: now 1785970078711107 us; error 40 us; skew 500 ppm
I20260805 22:47:58.711546 15996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:58 dist-test-slave-7lbf krb5kdc[15978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970078, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:58.715049 15996 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:47:58.716651 15996 webserver.cc:460] Webserver started at http://127.4.28.62:37365/ using document root <none> and password file <none>
I20260805 22:47:58.717407 15996 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:58.717541 15996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:58.717875 15996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:58.721048 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "9b7c9eb8476d4082b560ad84144b44ea"
format_stamp: "Formatted at 2026-08-05 22:47:58 on dist-test-slave-7lbf"
server_key: "efbdd8cde3fd11852029ca95e68721a3"
server_key_iv: "22baf62606e20bba69f6a9116521722e"
server_key_version: "encryptionkey@0"
I20260805 22:47:58.721954 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "9b7c9eb8476d4082b560ad84144b44ea"
format_stamp: "Formatted at 2026-08-05 22:47:58 on dist-test-slave-7lbf"
server_key: "efbdd8cde3fd11852029ca95e68721a3"
server_key_iv: "22baf62606e20bba69f6a9116521722e"
server_key_version: "encryptionkey@0"
I20260805 22:47:58.727670 15996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260805 22:47:58.731643 16016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:58.733225 15996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:47:58.733451 15996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "9b7c9eb8476d4082b560ad84144b44ea"
format_stamp: "Formatted at 2026-08-05 22:47:58 on dist-test-slave-7lbf"
server_key: "efbdd8cde3fd11852029ca95e68721a3"
server_key_iv: "22baf62606e20bba69f6a9116521722e"
server_key_version: "encryptionkey@0"
I20260805 22:47:58.733737 15996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:58.763785 15996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:58.768853 15996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:58.769183 15996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:58.789984 15996 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:44295
I20260805 22:47:58.790091 16077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:44295 every 8 connection(s)
I20260805 22:47:58.792179 15996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:47:58.793656 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 15996
I20260805 22:47:58.793903 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:47:58.794564 4208 external_mini_cluster.cc:1428] Setting key c597f2e7c9d73baf0a03e0bfccad0b89
I20260805 22:47:58.798857 16078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:47:58 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:58.810509 16078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: Bootstrap starting.
I20260805 22:47:58.814169 16078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: Neither blocks nor log segments found. Creating new log.
I20260805 22:47:58.815306 16078 log.cc:824] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: Log is configured to *not* fsync() on all Append() calls
I20260805 22:47:58.818233 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:58.796832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.4.28.62:44295)
0805 22:47:58.797779 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:58.797793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:58.797847 (+ 54us) server_negotiation.cc:407] Connection header received
0805 22:47:58.799721 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:58.799750 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:58.800233 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:58.800655 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:58.802340 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:58.803874 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:58.805205 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:58.805970 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:58.806157 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:58.810414 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:58.810423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:58.810442 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:58.810496 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:58.813715 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:58.814536 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:58.814545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:58.814556 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:58.814754 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:58.815173 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:58.815177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:58.815179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:58.815781 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:58.816110 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:58.816635 (+ 525us) server_negotiation.cc:299] Negotiation successful
0805 22:47:58.817123 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":381,"threads_started":1}
I20260805 22:47:58.818595 16078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: No bootstrap required, opened a new log
I20260805 22:47:58.827353 16078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } }
I20260805 22:47:58.827788 16078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:47:58.827908 16078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7c9eb8476d4082b560ad84144b44ea, State: Initialized, Role: FOLLOWER
I20260805 22:47:58.828572 16078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } }
I20260805 22:47:58.828814 16078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:47:58.828948 16078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:47:58.829130 16078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:47:58.831709 16078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } }
I20260805 22:47:58.832299 16078 leader_election.cc:302] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b7c9eb8476d4082b560ad84144b44ea; no voters:
I20260805 22:47:58.832981 16078 leader_election.cc:288] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:47:58.833122 16084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:47:58.833894 16084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 1 LEADER]: Becoming Leader. State: Replica: 9b7c9eb8476d4082b560ad84144b44ea, State: Running, Role: LEADER
I20260805 22:47:58.834486 16084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } }
I20260805 22:47:58.835150 16078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:47:58.837234 16086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b7c9eb8476d4082b560ad84144b44ea. Latest consensus state: current_term: 1 leader_uuid: "9b7c9eb8476d4082b560ad84144b44ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } } }
I20260805 22:47:58.837658 16086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:58.838794 16085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b7c9eb8476d4082b560ad84144b44ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } } }
I20260805 22:47:58.839191 16085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: This master's current role is: LEADER
I20260805 22:47:58.840586 16093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:47:58.846855 16093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:47:58.857118 16093 catalog_manager.cc:1347] Generated new cluster ID: 33972f8a2d1d40fbbffefb7923843bff
I20260805 22:47:58.857259 16093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:47:58.877467 16093 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:47:58.879024 16093 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:47:58.886400 16093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: Generated new TSK 0
I20260805 22:47:58.887647 16093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:47:58.953424 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44295
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:59.313377 16118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:59.313755 16118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:59.313908 16118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:59.321637 16118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:59.321765 16118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:59.321921 16118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:47:59.332317 16118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:59.333889 16118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:59.335356 16118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:59.349637 16125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:59.349597 16124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:59.349566 16127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:59.350507 16118 server_base.cc:1034] running on GCE node
I20260805 22:47:59.351263 16118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:59.352296 16118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:59.353503 16118 hybrid_clock.cc:648] HybridClock initialized: now 1785970079353439 us; error 61 us; skew 500 ppm
I20260805 22:47:59.353866 16118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:59 dist-test-slave-7lbf krb5kdc[15978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970079, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:59.357134 16118 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:47:59.358775 16118 webserver.cc:460] Webserver started at http://127.4.28.1:43959/ using document root <none> and password file <none>
I20260805 22:47:59.359499 16118 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:59.359611 16118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:59.359907 16118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:59.363109 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "205dbda6f88e4381b2c519030efab4db"
format_stamp: "Formatted at 2026-08-05 22:47:59 on dist-test-slave-7lbf"
server_key: "77b72ffd158febfe2027b40d020a7d5b"
server_key_iv: "9f867adf70bbc9cd067e1dcc1df00c1a"
server_key_version: "encryptionkey@0"
I20260805 22:47:59.363933 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "205dbda6f88e4381b2c519030efab4db"
format_stamp: "Formatted at 2026-08-05 22:47:59 on dist-test-slave-7lbf"
server_key: "77b72ffd158febfe2027b40d020a7d5b"
server_key_iv: "9f867adf70bbc9cd067e1dcc1df00c1a"
server_key_version: "encryptionkey@0"
I20260805 22:47:59.369777 16118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260805 22:47:59.373895 16135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:59.375396 16118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:59.375581 16118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "205dbda6f88e4381b2c519030efab4db"
format_stamp: "Formatted at 2026-08-05 22:47:59 on dist-test-slave-7lbf"
server_key: "77b72ffd158febfe2027b40d020a7d5b"
server_key_iv: "9f867adf70bbc9cd067e1dcc1df00c1a"
server_key_version: "encryptionkey@0"
I20260805 22:47:59.375800 16118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:59.409782 16118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:59.414219 16118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:59.414520 16118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:59.415580 16118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:59.417024 16118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:59.417105 16118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:59.417239 16118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:59.417295 16118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:59.449872 16118 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:37583
I20260805 22:47:59.449911 16287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:37583 every 8 connection(s)
I20260805 22:47:59.451256 16118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:47:59.453109 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 16118
I20260805 22:47:59.453337 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:47:59.454078 4208 external_mini_cluster.cc:1428] Setting key 5d9d05d73fa5c1d40a0d9e2728205771
Aug 05 22:47:59 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970079, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:47:59.475116 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:59.454139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:43599 (local address 127.4.28.62:44295)
0805 22:47:59.454684 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0805 22:47:59.454690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:47:59.455944 (+ 1254us) server_negotiation.cc:407] Connection header received
0805 22:47:59.458616 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:47:59.458621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:47:59.458697 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:47:59.458798 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:47:59.460987 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:59.461785 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:59.464048 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:47:59.464462 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:47:59.464584 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:59.469013 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:47:59.469020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:47:59.469023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:47:59.469067 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:47:59.471725 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:59.472565 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:59.472572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:59.472575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:59.472710 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:47:59.473189 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:47:59.473192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:47:59.473195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:47:59.473491 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:47:59.473670 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0805 22:47:59.474476 (+ 806us) server_negotiation.cc:299] Negotiation successful
0805 22:47:59.474742 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":260,"threads_started":1}
I20260805 22:47:59.476259 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:47:59.454784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:43599)
0805 22:47:59.455743 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0805 22:47:59.455800 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0805 22:47:59.458248 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:47:59.459127 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:47:59.459141 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:47:59.459928 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:47:59.460778 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:59.460806 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:59.461974 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:59.461981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:59.463846 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:47:59.463865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:47:59.464736 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:47:59.464743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:47:59.465043 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:47:59.465835 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:47:59.465866 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:47:59.468717 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:47:59.471905 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:59.471918 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:59.471935 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:59.472420 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:59.472831 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:47:59.472836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:47:59.472841 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:47:59.473068 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:47:59.473648 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:47:59.473658 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:47:59.474059 (+ 401us) client_negotiation.cc:769] Sending connection context
0805 22:47:59.474342 (+ 283us) client_negotiation.cc:240] Negotiation successful
0805 22:47:59.474939 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":393,"threads_started":1}
I20260805 22:47:59.478225 16288 heartbeater.cc:347] Connected to a master server at 127.4.28.62:44295
I20260805 22:47:59.478580 16288 heartbeater.cc:464] Registering TS with master...
I20260805 22:47:59.479441 16288 heartbeater.cc:511] Master 127.4.28.62:44295 requested a full tablet report, sending...
I20260805 22:47:59.482452 16036 ts_manager.cc:194] Registered new tserver with Master: 205dbda6f88e4381b2c519030efab4db (127.4.28.1:37583)
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
I20260805 22:47:59.484500 16036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:43599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:47:59.521770 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44295
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:47:59.878239 16295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:47:59.878571 16295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:47:59.878702 16295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:47:59.886180 16295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:47:59.886327 16295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:47:59.886479 16295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:47:59.896593 16295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:47:59.898077 16295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:47:59.899507 16295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:47:59.913471 16301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:59.913450 16306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:47:59.913550 16302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:47:59.914765 16295 server_base.cc:1034] running on GCE node
I20260805 22:47:59.915467 16295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:47:59.916533 16295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:47:59.917753 16295 hybrid_clock.cc:648] HybridClock initialized: now 1785970079917699 us; error 43 us; skew 500 ppm
I20260805 22:47:59.918097 16295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:47:59 dist-test-slave-7lbf krb5kdc[15978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970079, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:47:59.921336 16295 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:47:59.923007 16295 webserver.cc:460] Webserver started at http://127.4.28.2:41161/ using document root <none> and password file <none>
I20260805 22:47:59.923753 16295 fs_manager.cc:359] Metadata directory not provided
I20260805 22:47:59.923883 16295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:47:59.924219 16295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:47:59.927384 16295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "22da9508e5504be8b9cc6913bc0afff0"
format_stamp: "Formatted at 2026-08-05 22:47:59 on dist-test-slave-7lbf"
server_key: "7974f76ee43db3cb5817a7fd51e5682f"
server_key_iv: "2f029faa18ca06f3d5a9c9b1ac745622"
server_key_version: "encryptionkey@0"
I20260805 22:47:59.928220 16295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "22da9508e5504be8b9cc6913bc0afff0"
format_stamp: "Formatted at 2026-08-05 22:47:59 on dist-test-slave-7lbf"
server_key: "7974f76ee43db3cb5817a7fd51e5682f"
server_key_iv: "2f029faa18ca06f3d5a9c9b1ac745622"
server_key_version: "encryptionkey@0"
I20260805 22:47:59.934070 16295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260805 22:47:59.938053 16317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:59.939663 16295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:47:59.939851 16295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "22da9508e5504be8b9cc6913bc0afff0"
format_stamp: "Formatted at 2026-08-05 22:47:59 on dist-test-slave-7lbf"
server_key: "7974f76ee43db3cb5817a7fd51e5682f"
server_key_iv: "2f029faa18ca06f3d5a9c9b1ac745622"
server_key_version: "encryptionkey@0"
I20260805 22:47:59.940073 16295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:47:59.961027 16295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:47:59.966439 16295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:47:59.966799 16295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:47:59.967923 16295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:47:59.969599 16295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:47:59.969692 16295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:47:59.969825 16295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:47:59.969894 16295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:00.003770 16295 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:34689
I20260805 22:48:00.003808 16465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:34689 every 8 connection(s)
I20260805 22:48:00.005189 16295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:48:00.011793 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 16295
I20260805 22:48:00.012038 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:48:00.012487 4208 external_mini_cluster.cc:1428] Setting key 535edd44ce1799e1723d8dd77bcf4205
Aug 05 22:48:00 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970079, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:48:00.027280 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:00.008115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:44033 (local address 127.4.28.62:44295)
0805 22:48:00.008599 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:00.008604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:00.009583 (+ 979us) server_negotiation.cc:407] Connection header received
0805 22:48:00.011548 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:00.011555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:00.011647 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:00.011750 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:00.013719 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.014536 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.016257 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.016728 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.016848 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:00.020992 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:00.020999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:00.021003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:00.021047 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:00.023740 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:00.024644 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:00.024649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:00.024652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:00.024801 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:00.025304 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:00.025308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:00.025311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:00.025665 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:00.025810 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:00.026624 (+ 814us) server_negotiation.cc:299] Negotiation successful
0805 22:48:00.026878 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
I20260805 22:48:00.028581 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:00.008622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:44033)
0805 22:48:00.009404 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:00.009444 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:00.011213 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:00.012072 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:00.012083 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:00.012726 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:00.013483 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:00.013539 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.014707 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.014711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:00.016079 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:00.016096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.017020 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.017030 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:00.017278 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:00.018140 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:00.018173 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:00.020662 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:00.023937 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:00.023946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:00.023970 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:00.024470 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:00.024930 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:00.024934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:00.024937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:00.025170 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:00.025805 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:00.025813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:00.026226 (+ 413us) client_negotiation.cc:769] Sending connection context
0805 22:48:00.026494 (+ 268us) client_negotiation.cc:240] Negotiation successful
0805 22:48:00.027116 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":305,"threads_started":1}
I20260805 22:48:00.030303 16466 heartbeater.cc:347] Connected to a master server at 127.4.28.62:44295
I20260805 22:48:00.030650 16466 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:00.031445 16466 heartbeater.cc:511] Master 127.4.28.62:44295 requested a full tablet report, sending...
I20260805 22:48:00.033066 16036 ts_manager.cc:194] Registered new tserver with Master: 22da9508e5504be8b9cc6913bc0afff0 (127.4.28.2:34689)
I20260805 22:48:00.034019 16036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:44033
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:48:00.077742 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44295
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:48:00.439253 16474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:00.439585 16474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:00.439735 16474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:00.447463 16474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:00.447615 16474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:00.447768 16474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:48:00.458272 16474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:00.459755 16474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:00.461206 16474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:00.475402 16481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:00.475402 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:00.475975 16486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:00.476300 16474 server_base.cc:1034] running on GCE node
I20260805 22:48:00.476981 16474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:00.478085 16474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:00.479310 16474 hybrid_clock.cc:648] HybridClock initialized: now 1785970080479256 us; error 49 us; skew 500 ppm
I20260805 22:48:00.479723 16474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:48:00 dist-test-slave-7lbf krb5kdc[15978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970080, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:48:00.483143 16474 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:48:00.484824 16474 webserver.cc:460] Webserver started at http://127.4.28.3:43491/ using document root <none> and password file <none>
I20260805 22:48:00.485626 16474 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:00.485764 16474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:00.486102 16474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:00.488071 16288 heartbeater.cc:503] Master 127.4.28.62:44295 was elected leader, sending a full tablet report...
I20260805 22:48:00.489532 16474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "7785fece207e4405aa6396b7182c031d"
format_stamp: "Formatted at 2026-08-05 22:48:00 on dist-test-slave-7lbf"
server_key: "1ec30bc49f31d319561c8e9d02c8e9a6"
server_key_iv: "8f334bbe2d43a47d02dd97098bb216c5"
server_key_version: "encryptionkey@0"
I20260805 22:48:00.490496 16474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "7785fece207e4405aa6396b7182c031d"
format_stamp: "Formatted at 2026-08-05 22:48:00 on dist-test-slave-7lbf"
server_key: "1ec30bc49f31d319561c8e9d02c8e9a6"
server_key_iv: "8f334bbe2d43a47d02dd97098bb216c5"
server_key_version: "encryptionkey@0"
I20260805 22:48:00.496594 16474 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260805 22:48:00.500881 16496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:00.502578 16474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260805 22:48:00.502784 16474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "7785fece207e4405aa6396b7182c031d"
format_stamp: "Formatted at 2026-08-05 22:48:00 on dist-test-slave-7lbf"
server_key: "1ec30bc49f31d319561c8e9d02c8e9a6"
server_key_iv: "8f334bbe2d43a47d02dd97098bb216c5"
server_key_version: "encryptionkey@0"
I20260805 22:48:00.503012 16474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:00.535451 16474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:00.540164 16474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:00.540488 16474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:00.541628 16474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:00.543226 16474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:00.543316 16474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:00.543448 16474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:00.543514 16474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:00.576591 16474 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:32957
I20260805 22:48:00.576642 16646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:32957 every 8 connection(s)
I20260805 22:48:00.578418 16474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:48:00.587982 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 16474
I20260805 22:48:00.588176 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:48:00.588632 4208 external_mini_cluster.cc:1428] Setting key 34e921eeb51bf9337c36a4b728e2c38c
Aug 05 22:48:00 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970080, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:48:00.600057 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:00.581240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:46393 (local address 127.4.28.62:44295)
0805 22:48:00.581762 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:00.581767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:00.582884 (+ 1117us) server_negotiation.cc:407] Connection header received
0805 22:48:00.584917 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:00.584923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:00.585046 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:00.585153 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:00.587074 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.587886 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.589712 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.590159 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.590278 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:00.594106 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:00.594112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:00.594115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:00.594154 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:00.596633 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:00.597463 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:00.597467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:00.597470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:00.597659 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:00.598136 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:00.598140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:00.598143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:00.598438 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:00.598581 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:00.599382 (+ 801us) server_negotiation.cc:299] Negotiation successful
0805 22:48:00.599645 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":256,"threads_started":1}
I20260805 22:48:00.601197 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:00.581917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:46393)
0805 22:48:00.582701 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:00.582746 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:00.584556 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:00.585489 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:00.585499 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:00.586175 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:00.586841 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:00.586863 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.588074 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.588080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:00.589548 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:00.589568 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.590413 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.590422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:00.590702 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:00.591399 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:00.591424 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:00.593836 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:00.596810 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:00.596820 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:00.596838 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:00.597317 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:00.597783 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:00.597788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:00.597790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:00.598002 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:00.598579 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:00.598589 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:00.598972 (+ 383us) client_negotiation.cc:769] Sending connection context
0805 22:48:00.599239 (+ 267us) client_negotiation.cc:240] Negotiation successful
0805 22:48:00.599857 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":301,"threads_started":1}
I20260805 22:48:00.602912 16647 heartbeater.cc:347] Connected to a master server at 127.4.28.62:44295
I20260805 22:48:00.603292 16647 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:00.604187 16647 heartbeater.cc:511] Master 127.4.28.62:44295 requested a full tablet report, sending...
I20260805 22:48:00.605805 16036 ts_manager.cc:194] Registered new tserver with Master: 7785fece207e4405aa6396b7182c031d (127.4.28.3:32957)
I20260805 22:48:00.606643 16036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:46393
I20260805 22:48:00.617606 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab.
Entry for principal oryx/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab.
WARNING: no policy specified for oryx/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab.
Entry for principal oryx/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab.
WARNING: no policy specified for oryx/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab.
Entry for principal oryx/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab.
I20260805 22:48:00.800850 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 16118
Aug 05 22:48:00 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:48:00.818990 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:00.802222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.4.28.1:37583)
0805 22:48:00.803032 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:00.803043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:00.803470 (+ 427us) server_negotiation.cc:407] Connection header received
0805 22:48:00.803669 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:00.803679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:00.803938 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:00.804102 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:00.805615 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.806664 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.808075 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.808721 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.808840 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:00.813144 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:00.813160 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:00.813180 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:00.813227 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:00.816226 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:00.817074 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:00.817083 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:00.817085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:00.817230 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:00.817566 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:00.817571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:00.817574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:00.817952 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:00.818226 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:00.818463 (+ 237us) server_negotiation.cc:299] Negotiation successful
0805 22:48:00.818686 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":376,"threads_started":1}
I20260805 22:48:00.820477 16281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:00.904539 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 16118
I20260805 22:48:00.934757 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 16295
Aug 05 22:48:00 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:48:00.954413 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:00.936267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.4.28.2:34689)
0805 22:48:00.937213 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:00.937224 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:00.937259 (+ 35us) server_negotiation.cc:407] Connection header received
0805 22:48:00.937351 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:00.937361 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:00.937832 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:00.938062 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:00.939622 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.941401 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.943035 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:00.944025 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:00.944197 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:00.948637 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:00.948650 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:00.948668 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:00.948718 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:00.951779 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:00.952499 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:00.952507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:00.952510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:00.952644 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:00.952940 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:00.952943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:00.952945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:00.953306 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:00.953601 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:00.953870 (+ 269us) server_negotiation.cc:299] Negotiation successful
0805 22:48:00.954117 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":538,"threads_started":1}
I20260805 22:48:00.955799 16460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:01.038412 16466 heartbeater.cc:503] Master 127.4.28.62:44295 was elected leader, sending a full tablet report...
I20260805 22:48:01.040409 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 16295
I20260805 22:48:01.064397 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 16474
Aug 05 22:48:01 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:48:01.081959 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:01.065563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45032 (local address 127.4.28.3:32957)
0805 22:48:01.066170 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:01.066178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:01.066211 (+ 33us) server_negotiation.cc:407] Connection header received
0805 22:48:01.066278 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:01.066285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:01.066513 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:01.066672 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:01.068106 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:01.069145 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:01.070722 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:01.071330 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:01.071457 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:01.076060 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:01.076078 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:01.076096 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:01.076137 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:01.078996 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:01.079721 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:01.079731 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:01.079733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:01.079868 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:01.080277 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:01.080282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:01.080286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:01.080666 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:01.080987 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:01.081302 (+ 315us) server_negotiation.cc:299] Negotiation successful
0805 22:48:01.081611 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":246,"threads_started":1}
I20260805 22:48:01.083372 16639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:01.166961 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 16474
I20260805 22:48:01.185266 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 15996
I20260805 22:48:01.186672 16072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:01.292287 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 15996
I20260805 22:48:01.319116 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44295
--webserver_interface=127.4.28.62
--webserver_port=37365
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:44295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:48:01.676184 16669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:01.676527 16669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:01.676668 16669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:01.684207 16669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:48:01.684345 16669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:01.684396 16669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:01.684484 16669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:48:01.684548 16669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:48:01.695071 16669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:44295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=37365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:01.696600 16669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:01.698148 16669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:01.711000 16675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:01.710978 16678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:01.711086 16676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:01.711917 16669 server_base.cc:1034] running on GCE node
I20260805 22:48:01.712818 16669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:01.714475 16669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:01.715705 16669 hybrid_clock.cc:648] HybridClock initialized: now 1785970081715639 us; error 51 us; skew 500 ppm
I20260805 22:48:01.716075 16669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:48:01 dist-test-slave-7lbf krb5kdc[15978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970081, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:48:01.719314 16669 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:48:01.720810 16669 webserver.cc:460] Webserver started at http://127.4.28.62:37365/ using document root <none> and password file <none>
I20260805 22:48:01.721540 16669 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:01.721678 16669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:01.727649 16669 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:48:01.731026 16686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:01.732612 16669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260805 22:48:01.732777 16669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "9b7c9eb8476d4082b560ad84144b44ea"
format_stamp: "Formatted at 2026-08-05 22:47:58 on dist-test-slave-7lbf"
server_key: "efbdd8cde3fd11852029ca95e68721a3"
server_key_iv: "22baf62606e20bba69f6a9116521722e"
server_key_version: "encryptionkey@0"
I20260805 22:48:01.733903 16669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:01.772617 16669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:01.777189 16669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:01.777447 16669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:01.796274 16669 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:44295
I20260805 22:48:01.796324 16758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:44295 every 8 connection(s)
I20260805 22:48:01.797959 16669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:48:01.799455 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 16669
I20260805 22:48:01.800515 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:37583
--local_ip_for_outbound_sockets=127.4.28.1
--tserver_master_addrs=127.4.28.62:44295
--webserver_port=43959
--webserver_interface=127.4.28.1
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.28.1
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:48:01.807756 16759 sys_catalog.cc:263] Verifying existing consensus state
I20260805 22:48:01.812602 16759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: Bootstrap starting.
I20260805 22:48:01.838996 16759 log.cc:824] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: Log is configured to *not* fsync() on all Append() calls
I20260805 22:48:01.851672 16759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:48:01.852384 16759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: Bootstrap complete.
I20260805 22:48:01.865995 16759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } }
I20260805 22:48:01.867331 16759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7c9eb8476d4082b560ad84144b44ea, State: Initialized, Role: FOLLOWER
I20260805 22:48:01.868224 16759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } }
I20260805 22:48:01.868582 16759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:48:01.868719 16759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:48:01.868911 16759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:48:01.875231 16759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } }
I20260805 22:48:01.875962 16759 leader_election.cc:302] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b7c9eb8476d4082b560ad84144b44ea; no voters:
I20260805 22:48:01.876677 16759 leader_election.cc:288] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260805 22:48:01.876848 16763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:48:01.878043 16763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [term 2 LEADER]: Becoming Leader. State: Replica: 9b7c9eb8476d4082b560ad84144b44ea, State: Running, Role: LEADER
I20260805 22:48:01.878698 16763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } }
I20260805 22:48:01.879071 16759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:48:01.881597 16764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b7c9eb8476d4082b560ad84144b44ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } } }
I20260805 22:48:01.881593 16765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b7c9eb8476d4082b560ad84144b44ea. Latest consensus state: current_term: 2 leader_uuid: "9b7c9eb8476d4082b560ad84144b44ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7c9eb8476d4082b560ad84144b44ea" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44295 } } }
I20260805 22:48:01.882089 16764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: This master's current role is: LEADER
I20260805 22:48:01.882094 16765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea [sys.catalog]: This master's current role is: LEADER
I20260805 22:48:01.897118 16771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:48:01.903518 16771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:48:01.905138 16771 catalog_manager.cc:1259] Loaded cluster ID: 33972f8a2d1d40fbbffefb7923843bff
I20260805 22:48:01.905331 16771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:48:01.909711 16771 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:48:01.911666 16771 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b7c9eb8476d4082b560ad84144b44ea: Loaded TSK: 0
I20260805 22:48:01.913573 16771 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 22:48:02.206262 16761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:02.206578 16761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:02.206701 16761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:02.214493 16761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:02.214630 16761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:02.214778 16761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:48:02.226004 16761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.1.keytab
--principal=oryx/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:37583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=43959
--tserver_master_addrs=127.4.28.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:02.227479 16761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:02.228948 16761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:02.244912 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:02.245422 16796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:02.245476 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:02.246004 16761 server_base.cc:1034] running on GCE node
I20260805 22:48:02.246742 16761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:02.247885 16761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:02.249114 16761 hybrid_clock.cc:648] HybridClock initialized: now 1785970082249056 us; error 50 us; skew 500 ppm
I20260805 22:48:02.249466 16761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970082, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:48:02.252911 16761 init.cc:377] Logged in from keytab as oryx/127.4.28.1@KRBTEST.COM (short username oryx)
I20260805 22:48:02.254683 16761 webserver.cc:460] Webserver started at http://127.4.28.1:43959/ using document root <none> and password file <none>
I20260805 22:48:02.255430 16761 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:02.255573 16761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:02.261770 16761 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260805 22:48:02.264873 16805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:02.266497 16761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:48:02.266678 16761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "205dbda6f88e4381b2c519030efab4db"
format_stamp: "Formatted at 2026-08-05 22:47:59 on dist-test-slave-7lbf"
server_key: "77b72ffd158febfe2027b40d020a7d5b"
server_key_iv: "9f867adf70bbc9cd067e1dcc1df00c1a"
server_key_version: "encryptionkey@0"
I20260805 22:48:02.267728 16761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:02.302630 16761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:02.307345 16761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:02.307617 16761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:02.308702 16761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:02.310258 16761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:02.310353 16761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:02.310465 16761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:02.310561 16761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:02.343984 16761 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:37583
I20260805 22:48:02.344099 16948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:37583 every 8 connection(s)
I20260805 22:48:02.345633 16761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:48:02.351402 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 16761
I20260805 22:48:02.352186 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:34689
--local_ip_for_outbound_sockets=127.4.28.2
--tserver_master_addrs=127.4.28.62:44295
--webserver_port=41161
--webserver_interface=127.4.28.2
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.28.2
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
W20260805 22:48:02.368459 16952 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:02.348954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:60769 (local address 127.4.28.62:44295)
0805 22:48:02.349872 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:02.349890 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:02.350146 (+ 256us) server_negotiation.cc:407] Connection header received
0805 22:48:02.353048 (+ 2902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:02.353064 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:02.353483 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:02.353887 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:02.356086 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.357441 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.359827 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.360771 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.360982 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.367409 (+ 6427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:60769: BlockingRecv error: recv got EOF from 127.4.28.1:60769 (error 108)
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":290,"threads_started":1}
I20260805 22:48:02.368484 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:02.349013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:60769)
0805 22:48:02.349947 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:02.350010 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:02.352360 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:02.354153 (+ 1793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:02.354164 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:02.354947 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:02.355798 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.355827 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.357690 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.357696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.359629 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.359651 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.361068 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.361075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.361365 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.362185 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:02.362218 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:02.367020 (+ 4802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":404,"threads_started":1}
W20260805 22:48:02.369342 16949 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:44295 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.28.62:44295: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:02.378974 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:02.370165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:37539)
0805 22:48:02.370359 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:02.370382 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:02.370494 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:02.370799 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:02.370803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:02.371197 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:02.371395 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.371408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.372813 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.372818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.373845 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.373861 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.374759 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.374765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.374994 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.375624 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:02.375645 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:02.378741 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260805 22:48:02.379344 16952 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:02.370310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:37539 (local address 127.4.28.62:44295)
0805 22:48:02.370486 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:02.370492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:02.370515 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:48:02.370598 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:02.370603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:02.370677 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:02.370777 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:02.371572 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.372636 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.374013 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.374561 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.374731 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.379094 (+ 4363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:37539: BlockingRecv error: recv got EOF from 127.4.28.1:37539 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:02.388527 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:02.379816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:38895)
0805 22:48:02.380006 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:02.380027 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:02.380134 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:02.380580 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:02.380584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:02.380926 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:02.381098 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.381107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.382613 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.382618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.383593 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.383608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.384549 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.384554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.384691 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.385339 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:02.385359 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:02.388291 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260805 22:48:02.388876 16952 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:02.379927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:38895 (local address 127.4.28.62:44295)
0805 22:48:02.380124 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:02.380131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:02.380154 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:48:02.380280 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:02.380285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:02.380395 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:02.380509 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:02.381252 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.382437 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.383760 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.384396 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.384542 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.388593 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:38895: BlockingRecv error: recv got EOF from 127.4.28.1:38895 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260805 22:48:02.388918 16949 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 22:48:02.716707 16954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:02.717046 16954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:02.717146 16954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:02.724570 16954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:02.724701 16954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:02.724841 16954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:48:02.735195 16954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.2.keytab
--principal=oryx/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:34689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=41161
--tserver_master_addrs=127.4.28.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:02.736593 16954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:02.737941 16954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:02.751506 16960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:02.751556 16961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:02.751457 16963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:02.752786 16954 server_base.cc:1034] running on GCE node
I20260805 22:48:02.753484 16954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:02.754606 16954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:02.755829 16954 hybrid_clock.cc:648] HybridClock initialized: now 1785970082755769 us; error 47 us; skew 500 ppm
I20260805 22:48:02.756196 16954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970082, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:48:02.759691 16954 init.cc:377] Logged in from keytab as oryx/127.4.28.2@KRBTEST.COM (short username oryx)
I20260805 22:48:02.761394 16954 webserver.cc:460] Webserver started at http://127.4.28.2:41161/ using document root <none> and password file <none>
I20260805 22:48:02.762189 16954 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:02.762324 16954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:02.768676 16954 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 22:48:02.771935 16972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:02.773473 16954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:48:02.773726 16954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "22da9508e5504be8b9cc6913bc0afff0"
format_stamp: "Formatted at 2026-08-05 22:47:59 on dist-test-slave-7lbf"
server_key: "7974f76ee43db3cb5817a7fd51e5682f"
server_key_iv: "2f029faa18ca06f3d5a9c9b1ac745622"
server_key_version: "encryptionkey@0"
I20260805 22:48:02.774761 16954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:02.797865 16954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:02.802559 16954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:02.802830 16954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:02.803874 16954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:02.805344 16954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:02.805433 16954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:02.805581 16954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:02.805657 16954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:02.837764 16954 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:34689
I20260805 22:48:02.837838 17126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:34689 every 8 connection(s)
I20260805 22:48:02.839500 16954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:48:02.847767 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 16954
I20260805 22:48:02.848376 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:32957
--local_ip_for_outbound_sockets=127.4.28.3
--tserver_master_addrs=127.4.28.62:44295
--webserver_port=43491
--webserver_interface=127.4.28.3
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.28.3
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
W20260805 22:48:02.858198 16952 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:02.842310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:50087 (local address 127.4.28.62:44295)
0805 22:48:02.842540 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:02.842547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:02.843755 (+ 1208us) server_negotiation.cc:407] Connection header received
0805 22:48:02.845649 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:02.845653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:02.845732 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:02.845831 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:02.847729 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.848547 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.851096 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.851585 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.851706 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.857881 (+ 6175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:50087: BlockingRecv error: recv got EOF from 127.4.28.2:50087 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260805 22:48:02.859105 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:02.842802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:50087)
0805 22:48:02.843571 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:02.843613 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:02.845318 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:02.846123 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:02.846133 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:02.846723 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:02.847510 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.847544 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.848763 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.848769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.850865 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.850887 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.851882 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.851889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.852207 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.852999 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:02.853037 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:02.857577 (+ 4540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":297,"threads_started":1}
W20260805 22:48:02.859928 17128 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:44295 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.28.62:44295: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:02.870023 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:02.860661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:59293)
0805 22:48:02.860866 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:02.860888 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:02.861035 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:02.861402 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:02.861407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:02.861782 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:02.861953 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.861962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.863508 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.863513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.864475 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.864493 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.865639 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.865644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.865787 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.866468 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:02.866487 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:02.869775 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260805 22:48:02.870641 16952 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:02.860825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:59293 (local address 127.4.28.62:44295)
0805 22:48:02.860981 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:02.860985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:02.861007 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:48:02.861161 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:02.861164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:02.861236 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:02.861348 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:02.862148 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.863310 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.864664 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.865421 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.865615 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.870342 (+ 4727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:59293: BlockingRecv error: recv got EOF from 127.4.28.2:59293 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:02 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:02.880153 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:02.870806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:60681)
0805 22:48:02.871061 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:02.871077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:02.871196 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:02.871952 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:02.871957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:02.872340 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:02.872521 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.872531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.873750 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.873754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.874704 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:02.874719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.875457 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.875463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:02.875650 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.876309 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:02.876326 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:02.879831 (+ 3505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135}
W20260805 22:48:02.880560 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:02.871001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:60681 (local address 127.4.28.62:44295)
0805 22:48:02.871627 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:02.871633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:02.871651 (+ 18us) server_negotiation.cc:407] Connection header received
0805 22:48:02.871705 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:02.871708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:02.871778 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:02.871877 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:02.872695 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.873591 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.874864 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:02.875295 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:02.875417 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:02.880185 (+ 4768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:60681: BlockingRecv error: recv got EOF from 127.4.28.2:60681 (error 108)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":283,"threads_started":1}
W20260805 22:48:02.880672 17128 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 22:48:03.221868 17131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:03.222218 17131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:03.222385 17131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:03.230213 17131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:03.230363 17131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:03.230556 17131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:48:03.241204 17131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/krb5kdc/oryx_127.4.28.3.keytab
--principal=oryx/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:32957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=43491
--tserver_master_addrs=127.4.28.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:03.242741 17131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:03.244254 17131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:03.259356 17137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:03.259428 17138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:03.259599 17140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:03.260872 17131 server_base.cc:1034] running on GCE node
I20260805 22:48:03.261649 17131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:03.262782 17131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:03.264043 17131 hybrid_clock.cc:648] HybridClock initialized: now 1785970083263938 us; error 72 us; skew 500 ppm
I20260805 22:48:03.264410 17131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970083, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:48:03.267848 17131 init.cc:377] Logged in from keytab as oryx/127.4.28.3@KRBTEST.COM (short username oryx)
I20260805 22:48:03.269484 17131 webserver.cc:460] Webserver started at http://127.4.28.3:43491/ using document root <none> and password file <none>
I20260805 22:48:03.270323 17131 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:03.270481 17131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:03.276592 17131 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 22:48:03.280057 17151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:03.281636 17131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:48:03.281827 17131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "7785fece207e4405aa6396b7182c031d"
format_stamp: "Formatted at 2026-08-05 22:48:00 on dist-test-slave-7lbf"
server_key: "1ec30bc49f31d319561c8e9d02c8e9a6"
server_key_iv: "8f334bbe2d43a47d02dd97098bb216c5"
server_key_version: "encryptionkey@0"
I20260805 22:48:03.282954 17131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:03.303901 17131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:03.308415 17131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:03.308663 17131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:03.309738 17131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:03.311221 17131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:03.311314 17131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:03.311409 17131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:03.311463 17131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:03.344796 17131 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:32957
I20260805 22:48:03.344853 17303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:32957 every 8 connection(s)
I20260805 22:48:03.346408 17131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:48:03.352948 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 17131
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
W20260805 22:48:03.366317 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:03.349615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:34685 (local address 127.4.28.62:44295)
0805 22:48:03.349800 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:03.349805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:03.351253 (+ 1448us) server_negotiation.cc:407] Connection header received
0805 22:48:03.353245 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:03.353252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:03.353352 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:03.353490 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:03.355643 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.356754 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.359160 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.359803 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.359933 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.366038 (+ 6105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:34685: BlockingRecv error: recv got EOF from 127.4.28.3:34685 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 22:48:03.367101 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:03.350109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:34685)
0805 22:48:03.350986 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:03.351047 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:03.352861 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:03.353901 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:03.353916 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:03.354763 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:03.355454 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.355475 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.357000 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.357006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.358978 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.358995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.360094 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.360102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.360446 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.361320 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:03.361396 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:03.365795 (+ 4399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":363,"threads_started":1}
W20260805 22:48:03.367933 17305 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:44295 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.28.62:44295: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
I20260805 22:48:03.371546 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:03.354144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.4.28.62:44295)
0805 22:48:03.354312 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:03.354317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:03.354761 (+ 444us) server_negotiation.cc:407] Connection header received
0805 22:48:03.354922 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:03.354927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:03.354994 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:03.355077 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:03.357001 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.358192 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.359921 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.360512 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.360668 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.363671 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:03.363682 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:03.363705 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:03.363765 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:03.367314 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:03.368474 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:03.368486 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:03.368496 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:03.368771 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:03.369268 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:03.369274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:03.369277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:03.369992 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:03.370323 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:03.370757 (+ 434us) server_negotiation.cc:299] Negotiation successful
0805 22:48:03.371152 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:03.378067 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:03.368586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:54203)
0805 22:48:03.368786 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:03.368807 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:03.368915 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:03.369240 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:03.369244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:03.369718 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:03.369908 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.369920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.371391 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.371398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.372558 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.372573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.373454 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.373460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.373748 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.374401 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:03.374429 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:03.377830 (+ 3401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260805 22:48:03.378388 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:03.368709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:54203 (local address 127.4.28.62:44295)
0805 22:48:03.368871 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:03.368876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:03.368895 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:48:03.369009 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:03.369021 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:03.369114 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:03.369222 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:03.370056 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.371245 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.372706 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.373254 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.373395 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.378093 (+ 4698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:54203: BlockingRecv error: recv got EOF from 127.4.28.3:54203 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:03.387282 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:03.378936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:56345)
0805 22:48:03.379211 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:03.379233 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:03.379385 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:03.379773 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:03.379778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:03.380174 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:03.380341 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.380350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.381444 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.381448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.382429 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.382441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.383154 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.383161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.383342 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.383959 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:03.383978 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:03.386997 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260805 22:48:03.387718 17305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 22:48:03.387781 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:03.378975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:56345 (local address 127.4.28.62:44295)
0805 22:48:03.379243 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:03.379248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:03.379305 (+ 57us) server_negotiation.cc:407] Connection header received
0805 22:48:03.379542 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:03.379545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:03.379639 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:03.379723 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:03.380517 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.381289 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.382579 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.383005 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.383137 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.387331 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:56345: BlockingRecv error: recv got EOF from 127.4.28.3:56345 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:03.398104 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:03.389567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:39975)
0805 22:48:03.390100 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:03.390118 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:03.390240 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:03.390555 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:03.390559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:03.391058 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:03.391265 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.391275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.392332 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.392336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.393345 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.393360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.394205 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.394211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.394351 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.394942 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:03.394968 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:03.397830 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":263,"threads_started":1}
W20260805 22:48:03.398357 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:03.389592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:39975 (local address 127.4.28.62:44295)
0805 22:48:03.389747 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:03.389751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:03.390182 (+ 431us) server_negotiation.cc:407] Connection header received
0805 22:48:03.390356 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:03.390360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:03.390424 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:03.390496 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:03.391386 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.392183 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.393506 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.394018 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.394125 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.398089 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:39975: BlockingRecv error: recv got EOF from 127.4.28.1:39975 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:03 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:03.890825 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:03.881382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:44441)
0805 22:48:03.882037 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:03.882058 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:03.882212 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:03.882623 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:03.882629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:03.883209 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:03.883475 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.883488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.884540 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.884546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.885701 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:03.885719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.886646 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.886653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:03.886807 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.887494 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:03.887516 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:03.890567 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":396,"threads_started":1}
W20260805 22:48:03.891250 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:03.881573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:44441 (local address 127.4.28.62:44295)
0805 22:48:03.881765 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:03.881772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:03.882145 (+ 373us) server_negotiation.cc:407] Connection header received
0805 22:48:03.882358 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:03.882362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:03.882447 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:03.882540 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:03.883611 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.884347 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.885916 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:03.886408 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:03.886525 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:03.890974 (+ 4449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:44441: BlockingRecv error: recv got EOF from 127.4.28.2:44441 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 05 22:48:04 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:04 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:04.398067 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:04.388428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:56795)
0805 22:48:04.389003 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:04.389021 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:04.389157 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:04.389561 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:04.389566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:04.390058 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:04.390275 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:04.390286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.391663 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.391669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:04.392836 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:04.392853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.393649 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.393654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:04.393791 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:04.394501 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:04.394521 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:04.397782 (+ 3261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":299,"threads_started":1}
W20260805 22:48:04.398350 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:04.388569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:56795 (local address 127.4.28.62:44295)
0805 22:48:04.388757 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:04.388762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:04.389108 (+ 346us) server_negotiation.cc:407] Connection header received
0805 22:48:04.389318 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:04.389324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:04.389400 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:04.389498 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:04.390460 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.391465 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.392977 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.393437 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.393585 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:04.398077 (+ 4492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:56795: BlockingRecv error: recv got EOF from 127.4.28.3:56795 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 05 22:48:04 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:04 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:04.408198 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:04.399240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:55255)
0805 22:48:04.399740 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:04.399756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:04.399907 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:04.400262 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:04.400267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:04.400710 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:04.400873 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:04.400885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.402066 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.402070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:04.403081 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:04.403097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.403997 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.404004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:04.404205 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:04.404924 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:04.404948 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:04.407878 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":266,"threads_started":1}
W20260805 22:48:04.408432 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:04.399302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:55255 (local address 127.4.28.62:44295)
0805 22:48:04.399483 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:04.399490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:04.399847 (+ 357us) server_negotiation.cc:407] Connection header received
0805 22:48:04.400048 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:04.400053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:04.400121 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:04.400206 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:04.401048 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.401889 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.403237 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.403819 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.403942 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:04.408176 (+ 4234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:55255: BlockingRecv error: recv got EOF from 127.4.28.1:55255 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 05 22:48:04 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:04 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:04.900698 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:04.891919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:47553)
0805 22:48:04.892508 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:04.892526 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:04.892656 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:04.893038 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:04.893043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:04.893457 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:04.893659 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:04.893669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.894790 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.894795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:04.895738 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:04.895752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.896531 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.896536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:04.896658 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:04.897292 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:04.897310 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:04.900346 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":321,"threads_started":1}
W20260805 22:48:04.900851 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:04.892097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:47553 (local address 127.4.28.62:44295)
0805 22:48:04.892279 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:04.892283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:04.892640 (+ 357us) server_negotiation.cc:407] Connection header received
0805 22:48:04.892764 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:04.892768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:04.892837 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:04.892938 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:04.893820 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.894633 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.895900 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:04.896351 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:04.896470 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:04.900574 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:47553: BlockingRecv error: recv got EOF from 127.4.28.2:47553 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 05 22:48:05 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:05 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:05.407943 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:05.399162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:59053)
0805 22:48:05.399713 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:05.399732 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:05.399862 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:05.400158 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:05.400161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:05.400581 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:05.400742 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:05.400752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.401977 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.401984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:05.402978 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:05.402995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.403812 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.403817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:05.403938 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:05.404562 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:05.404581 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:05.407604 (+ 3023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":294,"threads_started":1}
W20260805 22:48:05.408118 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:05.399348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:59053 (local address 127.4.28.62:44295)
0805 22:48:05.399526 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:05.399531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:05.399847 (+ 316us) server_negotiation.cc:407] Connection header received
0805 22:48:05.399970 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:05.399974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:05.400044 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:05.400130 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:05.400922 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.401788 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.403159 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.403642 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.403762 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:05.407887 (+ 4125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:59053: BlockingRecv error: recv got EOF from 127.4.28.3:59053 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 05 22:48:05 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:05 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:05.418386 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:05.409098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:59335)
0805 22:48:05.409649 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:05.409665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:05.409817 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:05.410348 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:05.410353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:05.410787 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:05.411108 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:05.411125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.412404 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.412408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:05.413345 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:05.413359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.414157 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.414163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:05.414331 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:05.414983 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:05.415002 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:05.418125 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":265,"threads_started":1}
W20260805 22:48:05.418646 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:05.409194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:59335 (local address 127.4.28.62:44295)
0805 22:48:05.409380 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:05.409386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:05.410015 (+ 629us) server_negotiation.cc:407] Connection header received
0805 22:48:05.410087 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:05.410091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:05.410161 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:05.410258 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:05.411270 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.412254 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.413503 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.413990 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.414099 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:05.418381 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:59335: BlockingRecv error: recv got EOF from 127.4.28.1:59335 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 05 22:48:05 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:05 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:05.910472 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:05.901863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:34423)
0805 22:48:05.902406 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:05.902423 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:05.902553 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:05.902890 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:05.902894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:05.903279 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:05.903463 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:05.903473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.904486 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.904490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:05.905440 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:05.905454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.906238 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.906243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:05.906379 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:05.906965 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:05.906981 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:05.910089 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":264,"threads_started":1}
W20260805 22:48:05.910665 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:05.901975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:34423 (local address 127.4.28.62:44295)
0805 22:48:05.902170 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:05.902175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:05.902496 (+ 321us) server_negotiation.cc:407] Connection header received
0805 22:48:05.902657 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:05.902661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:05.902741 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:05.902830 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:05.903605 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.904351 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.905637 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:05.906102 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:05.906207 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:05.910374 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:34423: BlockingRecv error: recv got EOF from 127.4.28.2:34423 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 05 22:48:06 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:06 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:06.418229 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:06.409011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:45889)
0805 22:48:06.409620 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:06.409638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:06.409777 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:06.410168 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:06.410172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:06.410570 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:06.410760 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:06.410770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.412020 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.412025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:06.412963 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:06.412979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.413961 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.413967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:06.414124 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:06.414758 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:06.414778 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:06.417882 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":305,"threads_started":1}
W20260805 22:48:06.418279 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:06.409138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:45889 (local address 127.4.28.62:44295)
0805 22:48:06.409321 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:06.409326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:06.409705 (+ 379us) server_negotiation.cc:407] Connection header received
0805 22:48:06.409896 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:06.409901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:06.410001 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:06.410093 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:06.410972 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.411839 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.413217 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.413761 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.413882 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:06.418077 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:45889: BlockingRecv error: recv got EOF from 127.4.28.3:45889 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 05 22:48:06 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:06 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:06.427605 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:06.419396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:57261)
0805 22:48:06.419971 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:06.419992 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:06.420133 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:06.420464 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:06.420469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:06.420836 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:06.421027 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:06.421037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.422114 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.422119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:06.423102 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:06.423115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.423847 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.423853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:06.424038 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:06.424630 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:06.424650 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:06.427357 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":279,"threads_started":1}
W20260805 22:48:06.427799 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:06.419447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:57261 (local address 127.4.28.62:44295)
0805 22:48:06.419623 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:06.419629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:06.420067 (+ 438us) server_negotiation.cc:407] Connection header received
0805 22:48:06.420265 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:06.420271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:06.420337 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:06.420426 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:06.421171 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.421977 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.423242 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.423700 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.423814 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:06.427587 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:57261: BlockingRecv error: recv got EOF from 127.4.28.1:57261 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 05 22:48:06 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:06 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:06.923364 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:06.911639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:44629)
0805 22:48:06.912154 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:06.912170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:06.912306 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:06.912694 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:06.912698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:06.913091 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:06.913250 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:06.913260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.914437 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.914441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:06.915446 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:06.915459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.916239 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.916245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:06.916439 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:06.919818 (+ 3379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:06.919840 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:06.923032 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":244,"threads_started":1}
W20260805 22:48:06.923570 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:06.911702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:44629 (local address 127.4.28.62:44295)
0805 22:48:06.911938 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:06.911945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:06.912255 (+ 310us) server_negotiation.cc:407] Connection header received
0805 22:48:06.912412 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:06.912416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:06.912522 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:06.912609 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:06.913392 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.914193 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.915595 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:06.916073 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:06.916194 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:06.923244 (+ 7050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:44629: BlockingRecv error: recv got EOF from 127.4.28.2:44629 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 05 22:48:07 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:07 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:07.427556 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:07.419316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:52401)
0805 22:48:07.419884 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:07.419902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:07.420024 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:07.420312 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:07.420315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:07.420701 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:07.420864 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:07.420873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.421871 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.421875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:07.422876 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:07.422891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.423669 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.423674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:07.423873 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:07.424491 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:07.424519 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:07.427304 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":300,"threads_started":1}
W20260805 22:48:07.427778 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:07.419391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:52401 (local address 127.4.28.62:44295)
0805 22:48:07.419556 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:07.419561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:07.419971 (+ 410us) server_negotiation.cc:407] Connection header received
0805 22:48:07.420117 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:07.420120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:07.420190 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:07.420275 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:07.420983 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.421750 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.423028 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.423485 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.423591 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:07.427567 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:52401: BlockingRecv error: recv got EOF from 127.4.28.3:52401 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 05 22:48:07 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:07 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:07.437963 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:07.428625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:35367)
0805 22:48:07.429408 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:07.429427 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:07.429611 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:07.430076 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:07.430081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:07.430531 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:07.430735 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:07.430746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.432107 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.432111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:07.433069 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:07.433084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.433962 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.433968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:07.434114 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:07.434696 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:07.434715 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:07.437663 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":514,"threads_started":1}
W20260805 22:48:07.438112 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:07.428703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:35367 (local address 127.4.28.62:44295)
0805 22:48:07.428976 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:07.428984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:07.429548 (+ 564us) server_negotiation.cc:407] Connection header received
0805 22:48:07.429768 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:07.429772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:07.429860 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:07.429989 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:07.430902 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.431941 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.433230 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.433773 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.433936 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:07.437899 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:35367: BlockingRecv error: recv got EOF from 127.4.28.1:35367 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 05 22:48:07 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:07 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:07.933548 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:07.924495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:60443)
0805 22:48:07.925081 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:07.925104 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:07.925262 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:07.925741 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:07.925745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:07.926250 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:07.926467 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:07.926478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.927676 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.927681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:07.928775 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:07.928791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.929586 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.929592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:07.929722 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:07.930312 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:07.930330 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:07.933231 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":278,"threads_started":1}
W20260805 22:48:07.933745 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:07.924792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:60443 (local address 127.4.28.62:44295)
0805 22:48:07.925061 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:07.925068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:07.925192 (+ 124us) server_negotiation.cc:407] Connection header received
0805 22:48:07.925415 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:07.925421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:07.925558 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:07.925660 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:07.926614 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.927509 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.928940 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:07.929377 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:07.929497 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:07.933496 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:60443: BlockingRecv error: recv got EOF from 127.4.28.2:60443 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Aug 05 22:48:08 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:08 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:08.437808 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:08.428782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:60725)
0805 22:48:08.429428 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:08.429445 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:08.429619 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:08.429966 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:08.429971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:08.430411 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:08.430602 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:08.430612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.431633 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.431639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:08.432700 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:08.432716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.433499 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.433504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:08.433722 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:08.434386 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:08.434405 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:08.437465 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":343,"threads_started":1}
W20260805 22:48:08.438068 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:08.429014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:60725 (local address 127.4.28.62:44295)
0805 22:48:08.429290 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:08.429296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:08.429544 (+ 248us) server_negotiation.cc:407] Connection header received
0805 22:48:08.429751 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:08.429755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:08.429820 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:08.429908 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:08.430769 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.431476 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.432856 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.433337 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.433488 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:08.437767 (+ 4279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:60725: BlockingRecv error: recv got EOF from 127.4.28.3:60725 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 05 22:48:08 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:08 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:08.447911 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:08.438959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:41323)
0805 22:48:08.439698 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:08.439715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:08.439869 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:08.440279 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:08.440284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:08.440672 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:08.440834 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:08.440844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.441953 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.441957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:08.442865 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:08.442877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.443641 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.443647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:08.443816 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:08.444434 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:08.444458 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:08.447674 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":517,"threads_started":1}
W20260805 22:48:08.448101 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:08.439046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:41323 (local address 127.4.28.62:44295)
0805 22:48:08.439280 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:08.439288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:08.439802 (+ 514us) server_negotiation.cc:407] Connection header received
0805 22:48:08.440041 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:08.440046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:08.440125 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:08.440210 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:08.440994 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.441771 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.443022 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.443475 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.443590 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:08.447879 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:41323: BlockingRecv error: recv got EOF from 127.4.28.1:41323 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 05 22:48:08 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:08 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:08.943523 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:08.934620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:35745)
0805 22:48:08.935177 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:08.935194 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:08.935322 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:08.935679 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:08.935683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:08.936093 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:08.936261 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:08.936271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.937472 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.937477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:08.938552 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:08.938569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.939381 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.939387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:08.939591 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:08.940190 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:08.940209 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:08.943286 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
W20260805 22:48:08.943750 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:08.934776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:35745 (local address 127.4.28.62:44295)
0805 22:48:08.935011 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:08.935017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:08.935280 (+ 263us) server_negotiation.cc:407] Connection header received
0805 22:48:08.935431 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:08.935434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:08.935504 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:08.935598 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:08.936433 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.937276 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.938709 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:08.939211 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:08.939321 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:08.943528 (+ 4207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:35745: BlockingRecv error: recv got EOF from 127.4.28.2:35745 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 05 22:48:09 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:09 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:09.447651 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:09.438760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:54955)
0805 22:48:09.439367 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:09.439385 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:09.439527 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:09.439829 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:09.439832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:09.440238 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:09.440405 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:09.440414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.441604 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.441608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:09.442556 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:09.442571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.443469 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.443473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:09.443615 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:09.444269 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:09.444289 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:09.447327 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":363,"threads_started":1}
W20260805 22:48:09.447845 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:09.438941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:54955 (local address 127.4.28.62:44295)
0805 22:48:09.439161 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:09.439166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:09.439499 (+ 333us) server_negotiation.cc:407] Connection header received
0805 22:48:09.439630 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:09.439634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:09.439703 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:09.439792 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:09.440606 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.441428 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.442754 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.443304 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.443422 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:09.447601 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:54955: BlockingRecv error: recv got EOF from 127.4.28.3:54955 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 05 22:48:09 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:09 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:09.457428 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:09.448882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:32955)
0805 22:48:09.449568 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:09.449588 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:09.449708 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:09.449989 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:09.449992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:09.450409 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:09.450602 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:09.450612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.451617 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.451622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:09.452611 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:09.452625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.453414 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.453421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:09.453634 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:09.454224 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:09.454242 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:09.457167 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":281,"threads_started":1}
W20260805 22:48:09.457652 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:09.448944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:32955 (local address 127.4.28.62:44295)
0805 22:48:09.449109 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:09.449115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:09.449651 (+ 536us) server_negotiation.cc:407] Connection header received
0805 22:48:09.449810 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:09.449814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:09.449877 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:09.449946 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:09.450719 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.451431 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.452746 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.453245 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.453335 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:09.457371 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:32955: BlockingRecv error: recv got EOF from 127.4.28.1:32955 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 05 22:48:09 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:09 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:09.953023 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:09.944511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:45979)
0805 22:48:09.945125 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:09.945141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:09.945277 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:09.945627 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:09.945632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:09.946088 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:09.946275 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:09.946285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.947305 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.947309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:09.948235 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:09.948248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.948964 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.948970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:09.949137 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:09.949750 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:09.949769 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:09.952754 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":257,"threads_started":1}
W20260805 22:48:09.953241 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:09.944516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:45979 (local address 127.4.28.62:44295)
0805 22:48:09.944697 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:09.944704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:09.945226 (+ 522us) server_negotiation.cc:407] Connection header received
0805 22:48:09.945399 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:09.945403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:09.945470 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:09.945593 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:09.946426 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.947153 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.948383 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:09.948808 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:09.948920 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:09.952999 (+ 4079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:45979: BlockingRecv error: recv got EOF from 127.4.28.2:45979 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 05 22:48:10 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:10 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:10.457700 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:10.448774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:43651)
0805 22:48:10.449447 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:10.449465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:10.449637 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:10.449963 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:10.449966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:10.450355 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:10.450535 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:10.450562 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.451563 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.451567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:10.452514 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:10.452527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.453253 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.453259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:10.453425 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:10.454231 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:10.454254 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:10.457390 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":272,"threads_started":1}
W20260805 22:48:10.457897 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:10.448939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:43651 (local address 127.4.28.62:44295)
0805 22:48:10.449116 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:10.449121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:10.449545 (+ 424us) server_negotiation.cc:407] Connection header received
0805 22:48:10.449738 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:10.449742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:10.449807 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:10.449892 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:10.450670 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.451442 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.452663 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.453109 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.453230 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:10.457658 (+ 4428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:43651: BlockingRecv error: recv got EOF from 127.4.28.3:43651 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 05 22:48:10 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:10 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:10.467198 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:10.458470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:40067)
0805 22:48:10.458962 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:10.458979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:10.459103 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:10.459384 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:10.459388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:10.459768 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:10.459935 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:10.459945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.461069 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.461074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:10.462105 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:10.462123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.462924 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.462928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:10.463083 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:10.463666 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:10.463683 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:10.466838 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
W20260805 22:48:10.467387 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:10.458627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:40067 (local address 127.4.28.62:44295)
0805 22:48:10.458787 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:10.458791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:10.459056 (+ 265us) server_negotiation.cc:407] Connection header received
0805 22:48:10.459216 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:10.459220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:10.459282 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:10.459355 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:10.460117 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.460878 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.462262 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.462755 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.462873 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:10.467119 (+ 4246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:40067: BlockingRecv error: recv got EOF from 127.4.28.1:40067 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 05 22:48:10 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:10 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:10.963606 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:10.954234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:32883)
0805 22:48:10.954787 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:10.954807 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:10.954948 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:10.955317 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:10.955322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:10.955813 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:10.956062 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:10.956078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.957327 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.957332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:10.958567 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:10.958583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.959345 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.959352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:10.959518 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:10.960202 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:10.960220 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:10.963339 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":406,"threads_started":1}
W20260805 22:48:10.963979 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:10.954340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:32883 (local address 127.4.28.62:44295)
0805 22:48:10.954558 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:10.954564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:10.954992 (+ 428us) server_negotiation.cc:407] Connection header received
0805 22:48:10.955090 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:10.955095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:10.955178 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:10.955263 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:10.956239 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.957153 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.958711 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:10.959141 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:10.959255 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:10.963698 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:32883: BlockingRecv error: recv got EOF from 127.4.28.2:32883 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 05 22:48:11 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:11 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:11.467664 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:11.459000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:54787)
0805 22:48:11.459670 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:11.459688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:11.459833 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:11.460290 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:11.460296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:11.460818 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:11.460996 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:11.461007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.462033 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.462037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:11.462965 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:11.462978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.463708 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.463712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:11.463860 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:11.464472 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:11.464499 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:11.467383 (+ 2884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":273,"threads_started":1}
W20260805 22:48:11.467918 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:11.459123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:54787 (local address 127.4.28.62:44295)
0805 22:48:11.459332 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:11.459338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:11.459789 (+ 451us) server_negotiation.cc:407] Connection header received
0805 22:48:11.459995 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:11.460001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:11.460087 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:11.460205 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:11.461142 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.461906 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.463110 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.463557 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.463656 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:11.467669 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:54787: BlockingRecv error: recv got EOF from 127.4.28.3:54787 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 05 22:48:11 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:11 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:11.477317 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:11.468266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:54765)
0805 22:48:11.468762 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:11.468778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:11.468931 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:11.469261 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:11.469266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:11.469803 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:11.470068 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:11.470082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.471643 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.471649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:11.472701 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:11.472715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.473424 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.473431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:11.473644 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:11.474182 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:11.474200 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:11.477061 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":250,"threads_started":1}
W20260805 22:48:11.477596 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:11.468338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:54765 (local address 127.4.28.62:44295)
0805 22:48:11.468511 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:11.468517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:11.468859 (+ 342us) server_negotiation.cc:407] Connection header received
0805 22:48:11.469059 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:11.469063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:11.469127 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:11.469209 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:11.470274 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.471503 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.472854 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.473264 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.473373 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:11.477302 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:54765: BlockingRecv error: recv got EOF from 127.4.28.1:54765 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 05 22:48:11 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:11 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:11.976701 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:11.964617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:44369)
0805 22:48:11.965200 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:11.965217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:11.965338 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:11.965689 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:11.965692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:11.966088 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:11.966289 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:11.966304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.967359 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.967363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:11.968448 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:11.968465 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.969260 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.969266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:11.969418 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:11.972949 (+ 3531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:11.972972 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:11.976338 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":293,"threads_started":1}
W20260805 22:48:11.976801 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:11.964778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:44369 (local address 127.4.28.62:44295)
0805 22:48:11.965028 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:11.965033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:11.965300 (+ 267us) server_negotiation.cc:407] Connection header received
0805 22:48:11.965453 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:11.965457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:11.965557 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:11.965653 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:11.966474 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.967223 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.968628 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:11.969070 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:11.969172 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:11.976589 (+ 7417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:44369: BlockingRecv error: recv got EOF from 127.4.28.2:44369 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 05 22:48:12 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:12 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:12.478706 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:12.469031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:33885)
0805 22:48:12.469620 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:12.469638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:12.469785 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:12.470118 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:12.470122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:12.470525 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:12.470730 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:12.470741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.471984 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.471989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:12.473283 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:12.473300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.474076 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.474082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:12.474256 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:12.474852 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:12.474871 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:12.478191 (+ 3320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":306,"threads_started":1}
W20260805 22:48:12.478773 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:12.469150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:33885 (local address 127.4.28.62:44295)
0805 22:48:12.469328 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:12.469333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:12.469734 (+ 401us) server_negotiation.cc:407] Connection header received
0805 22:48:12.469907 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:12.469911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:12.469981 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:12.470071 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:12.470909 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.471828 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.473421 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.473910 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.474028 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:12.478500 (+ 4472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:33885: BlockingRecv error: recv got EOF from 127.4.28.3:33885 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 05 22:48:12 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:12 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:12.487450 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:12.478402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:38029)
0805 22:48:12.478950 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:12.478971 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:12.479107 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:12.479727 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:12.479732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:12.480093 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:12.480254 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:12.480264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.481329 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.481334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:12.482380 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:12.482395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.483152 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.483159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:12.483330 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:12.483928 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:12.483947 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:12.487112 (+ 3165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":513,"threads_started":1}
W20260805 22:48:12.487581 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:12.478722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:38029 (local address 127.4.28.62:44295)
0805 22:48:12.479039 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:12.479045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:12.479064 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:48:12.479433 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:12.479437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:12.479520 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:12.479634 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:12.480409 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.481171 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.482552 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.482990 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.483105 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:12.487307 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:38029: BlockingRecv error: recv got EOF from 127.4.28.1:38029 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":798,"threads_started":1}
Aug 05 22:48:12 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:12 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:12.986531 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:12.977707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:45137)
0805 22:48:12.978244 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:12.978262 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:12.978396 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:12.978732 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:12.978737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:12.979213 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:12.979398 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:12.979409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.980623 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.980628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:12.981622 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:12.981638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.982314 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.982319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:12.982453 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:12.983023 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:12.983040 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:12.986182 (+ 3142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":302,"threads_started":1}
W20260805 22:48:12.986727 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:12.978194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:45137 (local address 127.4.28.62:44295)
0805 22:48:12.978369 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:12.978374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:12.978393 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:48:12.978510 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:12.978514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:12.978600 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:12.978701 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:12.979569 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.980481 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.981746 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:12.982187 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:12.982284 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:12.986419 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:45137: BlockingRecv error: recv got EOF from 127.4.28.2:45137 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 05 22:48:13 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:13 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:13.489672 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:13.480244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:34651)
0805 22:48:13.481022 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:13.481040 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:13.481190 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:13.481587 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:13.481593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:13.482010 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:13.482169 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:13.482179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.483224 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.483229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:13.484196 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:13.484210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.485071 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.485078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:13.485250 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:13.485982 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:13.486002 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:13.489377 (+ 3375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":536,"threads_started":1}
W20260805 22:48:13.489988 17132 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:13.480384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:34651 (local address 127.4.28.62:44295)
0805 22:48:13.480616 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:13.480622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:13.481138 (+ 516us) server_negotiation.cc:407] Connection header received
0805 22:48:13.481316 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:13.481322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:13.481400 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:13.481490 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:13.482338 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.483064 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.484366 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.484843 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.484964 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:13.489731 (+ 4767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:34651: BlockingRecv error: recv got EOF from 127.4.28.3:34651 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 05 22:48:13 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:13 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:13.498862 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:13.488475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:59847)
0805 22:48:13.490353 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:13.490372 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:13.490515 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:13.490897 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:13.490902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:13.491348 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:13.491541 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:13.491552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.492835 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.492842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:13.493937 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:13.493953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.494756 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.494761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:13.494889 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:13.495483 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:13.495501 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:13.498605 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1755,"thread_start_us":1621,"threads_started":1}
W20260805 22:48:13.499271 17356 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:13.488614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:59847 (local address 127.4.28.62:44295)
0805 22:48:13.489273 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:13.489279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:13.490463 (+ 1184us) server_negotiation.cc:407] Connection header received
0805 22:48:13.490669 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:13.490675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:13.490744 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:13.490838 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:13.491712 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.492683 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.494078 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.494605 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.494708 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:13.498946 (+ 4238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:59847: BlockingRecv error: recv got EOF from 127.4.28.1:59847 (error 108)
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":353,"threads_started":1}
Aug 05 22:48:13 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:13 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:13.996456 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:13.987796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:34439)
0805 22:48:13.988383 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:13.988401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:13.988541 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:13.988930 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:13.988935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:13.989400 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:13.989623 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:13.989633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.990951 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.990956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:13.991914 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:13.991928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.992699 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.992704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:13.992842 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:13.993411 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:13.993429 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:13.996190 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":293,"threads_started":1}
W20260805 22:48:13.996603 17356 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:13.987905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:34439 (local address 127.4.28.62:44295)
0805 22:48:13.988127 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:13.988134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:13.988485 (+ 351us) server_negotiation.cc:407] Connection header received
0805 22:48:13.988695 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:13.988699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:13.988772 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:13.988863 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:13.989767 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.990752 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.992059 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:13.992487 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:13.992583 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:13.996409 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:34439: BlockingRecv error: recv got EOF from 127.4.28.2:34439 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 05 22:48:14 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:14 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:14.499739 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:14.490657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:50789)
0805 22:48:14.491185 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:14.491205 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:14.491355 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:14.491733 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:14.491737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:14.492199 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:14.492388 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:14.492398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:14.493479 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:14.493484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:14.494465 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:14.494480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:14.495292 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:14.495297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:14.495496 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:14.496180 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:14.496198 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:14.499382 (+ 3184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":274,"threads_started":1}
W20260805 22:48:14.499873 17356 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:14.490869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:50789 (local address 127.4.28.62:44295)
0805 22:48:14.491141 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:14.491147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:14.491282 (+ 135us) server_negotiation.cc:407] Connection header received
0805 22:48:14.491475 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:14.491478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:14.491561 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:14.491641 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:14.492543 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:14.493330 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:14.494620 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:14.495139 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:14.495265 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:14.499614 (+ 4349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:50789: BlockingRecv error: recv got EOF from 127.4.28.3:50789 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 05 22:48:14 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:14 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:14.507886 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:14.499981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:45679)
0805 22:48:14.500524 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:14.500541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:14.500659 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:14.500922 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:14.500926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:14.501304 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:14.501469 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:14.501479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:14.502476 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:14.502479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:14.503445 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:14.503459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:14.504147 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:14.504154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:14.504295 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:14.504837 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:14.504855 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:14.507567 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":297,"threads_started":1}
W20260805 22:48:14.508076 17356 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:14.500326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:45679 (local address 127.4.28.62:44295)
0805 22:48:14.500467 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:14.500471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:14.500606 (+ 135us) server_negotiation.cc:407] Connection header received
0805 22:48:14.500749 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:14.500752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:14.500813 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:14.500891 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:14.501623 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:14.502359 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:14.503563 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:14.503983 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:14.504074 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:14.507826 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:45679: BlockingRecv error: recv got EOF from 127.4.28.1:45679 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 05 22:48:15 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:15 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:15.006978 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:14.997681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:60465)
0805 22:48:14.998352 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:14.998369 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:14.998539 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:14.998901 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:14.998906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:14.999363 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:14.999576 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:14.999590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.000927 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.000933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:15.002146 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:15.002162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.002918 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.002923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:15.003068 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:15.003704 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:15.003725 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:15.006690 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":400,"threads_started":1}
W20260805 22:48:15.007208 17356 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:14.997750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:60465 (local address 127.4.28.62:44295)
0805 22:48:14.998078 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:14.998085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:14.998485 (+ 400us) server_negotiation.cc:407] Connection header received
0805 22:48:14.998681 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:14.998685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:14.998753 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:14.998842 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:14.999749 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.000746 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.002300 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.002763 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.002863 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:15.006976 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:60465: BlockingRecv error: recv got EOF from 127.4.28.2:60465 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 05 22:48:15 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:15 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:15.510954 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:15.500967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:56973)
0805 22:48:15.501553 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:15.501570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:15.501698 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:15.502006 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:15.502010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:15.502438 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:15.502607 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:15.502617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.503794 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.503799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:15.504802 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:15.504819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.505696 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.505702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:15.505884 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:15.506622 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:15.506649 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:15.510571 (+ 3922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":301,"threads_started":1}
W20260805 22:48:15.511113 17356 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:15.500978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:56973 (local address 127.4.28.62:44295)
0805 22:48:15.501183 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:15.501188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:15.501646 (+ 458us) server_negotiation.cc:407] Connection header received
0805 22:48:15.501798 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:15.501802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:15.501887 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:15.501974 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:15.502804 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.503604 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.504977 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.505463 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.505638 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:15.510846 (+ 5208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:56973: BlockingRecv error: recv got EOF from 127.4.28.3:56973 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 05 22:48:15 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:15 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:15.518741 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:15.509255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:60055)
0805 22:48:15.510036 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:15.510058 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:15.510196 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:15.510882 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:15.510887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:15.511640 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:15.511853 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:15.511865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.512947 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.512952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:15.513959 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:15.513974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.514737 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.514743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:15.514905 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:15.515494 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:15.515521 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:15.518385 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":493,"threads_started":1}
W20260805 22:48:15.519035 17364 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:15.509501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:60055 (local address 127.4.28.62:44295)
0805 22:48:15.510380 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:15.510386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:15.510406 (+ 20us) server_negotiation.cc:407] Connection header received
0805 22:48:15.510463 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:15.510468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:15.510558 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:15.510662 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:15.512019 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.512795 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.514112 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:15.514547 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:15.514662 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:15.518729 (+ 4067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:60055: BlockingRecv error: recv got EOF from 127.4.28.1:60055 (error 108)
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":527,"threads_started":1}
Aug 05 22:48:16 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:16 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:16.017148 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:16.008104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:51177)
0805 22:48:16.008686 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:16.008703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:16.008848 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:16.009223 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:16.009228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:16.009730 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:16.009907 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:16.009918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.011045 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.011049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:16.012155 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:16.012172 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.012909 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.012916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:16.013090 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:16.013765 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:16.013784 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:16.016851 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":297,"threads_started":1}
W20260805 22:48:16.017349 17364 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:16.008347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:51177 (local address 127.4.28.62:44295)
0805 22:48:16.008605 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:16.008612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:16.008785 (+ 173us) server_negotiation.cc:407] Connection header received
0805 22:48:16.008985 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:16.008991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:16.009068 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:16.009157 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:16.010074 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.010862 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.012321 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.012744 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.012865 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:16.017115 (+ 4250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:51177: BlockingRecv error: recv got EOF from 127.4.28.2:51177 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 05 22:48:16 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:16 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:16.521683 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:16.512320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:58061)
0805 22:48:16.512920 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:16.512937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:16.513089 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:16.513474 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:16.513480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:16.513983 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:16.514157 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:16.514167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.515214 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.515219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:16.516161 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:16.516173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.516945 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.516952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:16.517124 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:16.517791 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:16.517810 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:16.521322 (+ 3512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":322,"threads_started":1}
W20260805 22:48:16.521821 17364 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:16.512446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:58061 (local address 127.4.28.62:44295)
0805 22:48:16.512659 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:16.512666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:16.513032 (+ 366us) server_negotiation.cc:407] Connection header received
0805 22:48:16.513217 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:16.513223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:16.513307 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:16.513400 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:16.514323 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.515058 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.516313 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.516753 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.516868 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:16.521581 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:58061: BlockingRecv error: recv got EOF from 127.4.28.3:58061 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 05 22:48:16 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:16 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:16.528818 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:16.519783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:51221)
0805 22:48:16.520325 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:16.520346 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:16.520492 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:16.521557 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:16.521562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:16.522038 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:16.522241 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:16.522252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.523391 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.523395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:16.524360 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:16.524373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.525047 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.525051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:16.525181 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:16.525782 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:16.525801 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:16.528571 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":266,"threads_started":1}
W20260805 22:48:16.529122 17368 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:16.519843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:51221 (local address 127.4.28.62:44295)
0805 22:48:16.521008 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:16.521014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:16.521032 (+ 18us) server_negotiation.cc:407] Connection header received
0805 22:48:16.521088 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:16.521092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:16.521167 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:16.521282 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:16.522388 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.523217 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.524474 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:16.524911 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:16.525005 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:16.528816 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:51221: BlockingRecv error: recv got EOF from 127.4.28.1:51221 (error 108)
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":764,"threads_started":1}
Aug 05 22:48:17 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:17 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:17.031014 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:17.018499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:54565)
0805 22:48:17.019140 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:17.019158 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:17.019306 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:17.019771 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:17.019775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:17.020182 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:17.020379 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:17.020389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.021556 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.021561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:17.022848 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:17.022867 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.023633 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.023639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:17.023810 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:17.027434 (+ 3624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:17.027463 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:17.030756 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":352,"threads_started":1}
W20260805 22:48:17.031348 17368 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:17.018716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:54565 (local address 127.4.28.62:44295)
0805 22:48:17.018968 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:17.018974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:17.019255 (+ 281us) server_negotiation.cc:407] Connection header received
0805 22:48:17.019533 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:17.019537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:17.019614 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:17.019712 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:17.020542 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.021329 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.023002 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.023443 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.023597 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:17.031097 (+ 7500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:54565: BlockingRecv error: recv got EOF from 127.4.28.2:54565 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 05 22:48:17 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:17 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.3@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
W20260805 22:48:17.532620 17368 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:17.522957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:41897 (local address 127.4.28.62:44295)
0805 22:48:17.523205 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:17.523213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:17.523419 (+ 206us) server_negotiation.cc:407] Connection header received
0805 22:48:17.523619 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:17.523623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:17.523712 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:17.523812 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:17.524707 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.525543 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.526904 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.527395 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.527528 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:17.532269 (+ 4741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.3:41897: BlockingRecv error: recv got EOF from 127.4.28.3:41897 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260805 22:48:17.532891 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:17.522752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.3:41897)
0805 22:48:17.523298 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:17.523321 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:17.523481 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:17.523872 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:17.523878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:17.524325 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:17.524535 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:17.524546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.525713 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.525718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:17.526725 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:17.526741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.527563 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.527570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:17.527757 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:17.528408 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:17.528428 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:17.532576 (+ 4148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":277,"threads_started":1}
Aug 05 22:48:17 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:17 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.1@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:17.539870 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:17.530248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.1:37273)
0805 22:48:17.530958 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:17.530980 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:17.531124 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:17.531585 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:17.531592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:17.532409 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:17.532699 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:17.532717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.534031 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.534035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:17.535027 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:17.535042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.535809 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.535814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:17.535989 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:17.536581 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:17.536600 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:17.539520 (+ 2920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":441,"threads_started":1}
W20260805 22:48:17.540069 17372 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:17.530429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:37273 (local address 127.4.28.62:44295)
0805 22:48:17.531250 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:17.531256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:17.531272 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:48:17.531323 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:17.531327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:17.531400 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:17.531802 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:17.532918 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.533855 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.535196 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:17.535648 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:17.535753 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:17.539814 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.1:37273: BlockingRecv error: recv got EOF from 127.4.28.1:37273 (error 108)
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":544,"threads_started":1}
Aug 05 22:48:18 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
Aug 05 22:48:18 dist-test-slave-7lbf krb5kdc[15978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.28.2@KRBTEST.COM for oryx/127.4.28.62@KRBTEST.COM, Server not found in Kerberos database
I20260805 22:48:18.040738 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:18.032226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44295 (local address 127.4.28.2:34573)
0805 22:48:18.032738 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:18.032755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:18.032871 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:18.033248 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:18.033253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:18.033750 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:18.033935 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:18.033945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.034940 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.034944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:18.035895 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:18.035908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.036581 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.036585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:18.036705 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:18.037346 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:18.037366 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:18.040459 (+ 3093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.28.62:44295: Server oryx/127.4.28.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
W20260805 22:48:18.041025 17372 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:18.032289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:34573 (local address 127.4.28.62:44295)
0805 22:48:18.032458 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:18.032462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:18.032823 (+ 361us) server_negotiation.cc:407] Connection header received
0805 22:48:18.032991 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:18.032996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:18.033086 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:18.033215 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:18.034064 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.034808 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.036014 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.036445 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.036548 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:18.040781 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.28.2:34573: BlockingRecv error: recv got EOF from 127.4.28.2:34573 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 22:48:18.360296 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 16761
I20260805 22:48:18.373920 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:18.361600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.4.28.1:37583)
0805 22:48:18.362255 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:18.362263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:18.362295 (+ 32us) server_negotiation.cc:407] Connection header received
0805 22:48:18.362360 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:18.362366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:18.362648 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:18.362821 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:18.364614 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.365752 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.367159 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.367799 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.367911 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:18.370658 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:18.370669 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:18.370687 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:18.370730 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:18.373328 (+ 2598us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.28.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0805 22:48:18.373682 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43984: Request ticket server kudu/127.4.28.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":338,"threads_started":1}
W20260805 22:48:18.374035 17377 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43984: Request ticket server kudu/127.4.28.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260805 22:48:18.374270 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 16761: Not authorized: Client connection negotiation failed: client connection to 127.4.28.1:37583: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.28.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260805 22:48:18.374450 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 16761
I20260805 22:48:18.401124 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 16954
I20260805 22:48:18.415901 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:18.402388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.4.28.2:34689)
0805 22:48:18.403116 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:18.403124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:18.403158 (+ 34us) server_negotiation.cc:407] Connection header received
0805 22:48:18.403231 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:18.403238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:18.403482 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:18.403695 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:18.405206 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.406396 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.407771 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.408420 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.408783 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:18.411830 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:18.411863 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:18.411887 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:18.411995 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:18.415302 (+ 3307us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.28.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0805 22:48:18.415645 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58820: Request ticket server kudu/127.4.28.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":327,"threads_started":1}
W20260805 22:48:18.416015 17378 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58820: Request ticket server kudu/127.4.28.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260805 22:48:18.416278 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 16954: Not authorized: Client connection negotiation failed: client connection to 127.4.28.2:34689: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.28.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260805 22:48:18.416468 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 16954
I20260805 22:48:18.441560 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 17131
I20260805 22:48:18.455169 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:18.442799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.4.28.3:32957)
0805 22:48:18.443405 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:18.443414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:18.443445 (+ 31us) server_negotiation.cc:407] Connection header received
0805 22:48:18.443516 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:18.443523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:18.443758 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:18.443944 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:18.445442 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.446704 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.448235 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:18.448840 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:18.448954 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:18.451934 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:18.451946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:18.451963 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:18.452042 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:18.454597 (+ 2555us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.28.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0805 22:48:18.454904 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49968: Request ticket server kudu/127.4.28.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":285,"threads_started":1}
W20260805 22:48:18.455276 17379 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49968: Request ticket server kudu/127.4.28.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260805 22:48:18.455543 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 17131: Not authorized: Client connection negotiation failed: client connection to 127.4.28.3:32957: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.28.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260805 22:48:18.455684 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 17131
I20260805 22:48:18.475879 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 16669
I20260805 22:48:18.477320 16752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:18.599074 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 16669
2026-08-05T22:48:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21793 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-05T22:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:48:18Z Disabled control of system clock
I20260805 22:48:18.697381 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34919
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260805 22:48:19.065029 17386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:19.065367 17386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:19.065451 17386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:19.073194 17386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:48:19.073338 17386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:19.073375 17386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:19.073426 17386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:48:19.073474 17386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:48:19.083760 17386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:34919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:19.085156 17386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:19.087208 17386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:19.099716 17393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:19.099694 17394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:19.099784 17398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:19.100347 17386 server_base.cc:1034] running on GCE node
I20260805 22:48:19.101253 17386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:19.102854 17386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:19.104091 17386 hybrid_clock.cc:648] HybridClock initialized: now 1785970099104059 us; error 51 us; skew 500 ppm
I20260805 22:48:19.104499 17386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:48:19.107123 17386 webserver.cc:460] Webserver started at http://127.4.28.62:33365/ using document root <none> and password file <none>
I20260805 22:48:19.107882 17386 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:19.107986 17386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:19.108342 17386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:19.111635 17386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "3fc2ac10ffa7424887cd279d795ac495"
format_stamp: "Formatted at 2026-08-05 22:48:19 on dist-test-slave-7lbf"
server_key: "3fdf98b4cfe88661241b7e5fa7dae9d9"
server_key_iv: "57ac419703ae35fef65970f3ad8c59af"
server_key_version: "encryptionkey@0"
I20260805 22:48:19.112509 17386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "3fc2ac10ffa7424887cd279d795ac495"
format_stamp: "Formatted at 2026-08-05 22:48:19 on dist-test-slave-7lbf"
server_key: "3fdf98b4cfe88661241b7e5fa7dae9d9"
server_key_iv: "57ac419703ae35fef65970f3ad8c59af"
server_key_version: "encryptionkey@0"
I20260805 22:48:19.118525 17386 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260805 22:48:19.122938 17407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:19.124624 17386 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260805 22:48:19.124837 17386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "3fc2ac10ffa7424887cd279d795ac495"
format_stamp: "Formatted at 2026-08-05 22:48:19 on dist-test-slave-7lbf"
server_key: "3fdf98b4cfe88661241b7e5fa7dae9d9"
server_key_iv: "57ac419703ae35fef65970f3ad8c59af"
server_key_version: "encryptionkey@0"
I20260805 22:48:19.125121 17386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:19.154106 17386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:19.155035 17386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:19.155328 17386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:19.174125 17386 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:34919
I20260805 22:48:19.174146 17477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:34919 every 8 connection(s)
I20260805 22:48:19.175631 17386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:48:19.178193 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 17386
I20260805 22:48:19.178385 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:48:19.179064 4208 external_mini_cluster.cc:1428] Setting key 15f5b29ee5c2ac4b0e3154758df0c3f3
I20260805 22:48:19.180523 17480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:48:19.193395 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:19.181338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.4.28.62:34919)
0805 22:48:19.182484 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:19.182499 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:19.182547 (+ 48us) server_negotiation.cc:407] Connection header received
0805 22:48:19.184162 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:19.184186 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:19.184720 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:19.185184 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:19.185828 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:19.187231 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:19.188641 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:19.189338 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:19.189537 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:19.190810 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:19.190818 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:19.190836 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:19.191479 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:19.191573 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:19.191598 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:19.191708 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:19.191962 (+ 254us) server_negotiation.cc:299] Negotiation successful
0805 22:48:19.192325 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":370,"threads_started":1}
I20260805 22:48:19.193506 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495: Bootstrap starting.
I20260805 22:48:19.197464 17480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495: Neither blocks nor log segments found. Creating new log.
I20260805 22:48:19.198652 17480 log.cc:824] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495: Log is configured to *not* fsync() on all Append() calls
I20260805 22:48:19.202123 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495: No bootstrap required, opened a new log
I20260805 22:48:19.210281 17480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc2ac10ffa7424887cd279d795ac495" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34919 } }
I20260805 22:48:19.210727 17480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:48:19.210836 17480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc2ac10ffa7424887cd279d795ac495, State: Initialized, Role: FOLLOWER
I20260805 22:48:19.211462 17480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc2ac10ffa7424887cd279d795ac495" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34919 } }
I20260805 22:48:19.211685 17480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:48:19.211829 17480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:48:19.211982 17480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:48:19.214349 17480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc2ac10ffa7424887cd279d795ac495" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34919 } }
I20260805 22:48:19.214902 17480 leader_election.cc:302] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fc2ac10ffa7424887cd279d795ac495; no voters:
I20260805 22:48:19.215579 17480 leader_election.cc:288] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:48:19.215727 17485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:48:19.216455 17485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [term 1 LEADER]: Becoming Leader. State: Replica: 3fc2ac10ffa7424887cd279d795ac495, State: Running, Role: LEADER
I20260805 22:48:19.217128 17485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc2ac10ffa7424887cd279d795ac495" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34919 } }
I20260805 22:48:19.218009 17480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:48:19.220213 17486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fc2ac10ffa7424887cd279d795ac495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc2ac10ffa7424887cd279d795ac495" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34919 } } }
I20260805 22:48:19.220651 17486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [sys.catalog]: This master's current role is: LEADER
I20260805 22:48:19.221390 17487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fc2ac10ffa7424887cd279d795ac495. Latest consensus state: current_term: 1 leader_uuid: "3fc2ac10ffa7424887cd279d795ac495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc2ac10ffa7424887cd279d795ac495" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34919 } } }
I20260805 22:48:19.221869 17487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495 [sys.catalog]: This master's current role is: LEADER
I20260805 22:48:19.222239 17496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:48:19.228114 17496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:48:19.229708 17506 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260805 22:48:19.238281 17496 catalog_manager.cc:1347] Generated new cluster ID: 83fcf4725aed4ab6afc22c13aab84b39
I20260805 22:48:19.238413 17496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:48:19.248059 17496 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:48:19.250036 17496 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:48:19.262318 17496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fc2ac10ffa7424887cd279d795ac495: Generated new TSK 0
I20260805 22:48:19.263576 17496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:48:19.282814 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34919
--builtin_ntp_servers=127.4.28.20:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260805 22:48:19.651515 17511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:19.651845 17511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:19.651947 17511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:19.659965 17511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:19.660105 17511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:19.660259 17511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:48:19.670044 17511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:19.671494 17511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:19.673475 17511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:19.688349 17520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:19.688285 17517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:19.688383 17516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:19.689221 17511 server_base.cc:1034] running on GCE node
I20260805 22:48:19.689951 17511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:19.691076 17511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:19.692315 17511 hybrid_clock.cc:648] HybridClock initialized: now 1785970099692250 us; error 55 us; skew 500 ppm
I20260805 22:48:19.692710 17511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:48:19.695261 17511 webserver.cc:460] Webserver started at http://127.4.28.1:37265/ using document root <none> and password file <none>
I20260805 22:48:19.696023 17511 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:19.696136 17511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:19.696486 17511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:19.699640 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "b308231dab1f4727b4ba6bb73972631e"
format_stamp: "Formatted at 2026-08-05 22:48:19 on dist-test-slave-7lbf"
server_key: "74a023c0b308a521a2ec3714f9f85e2f"
server_key_iv: "5c9a47f885055b2317fe81b982a9f3c1"
server_key_version: "encryptionkey@0"
I20260805 22:48:19.700567 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "b308231dab1f4727b4ba6bb73972631e"
format_stamp: "Formatted at 2026-08-05 22:48:19 on dist-test-slave-7lbf"
server_key: "74a023c0b308a521a2ec3714f9f85e2f"
server_key_iv: "5c9a47f885055b2317fe81b982a9f3c1"
server_key_version: "encryptionkey@0"
I20260805 22:48:19.706262 17511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260805 22:48:19.709983 17528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:19.711524 17511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:48:19.711709 17511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "b308231dab1f4727b4ba6bb73972631e"
format_stamp: "Formatted at 2026-08-05 22:48:19 on dist-test-slave-7lbf"
server_key: "74a023c0b308a521a2ec3714f9f85e2f"
server_key_iv: "5c9a47f885055b2317fe81b982a9f3c1"
server_key_version: "encryptionkey@0"
I20260805 22:48:19.711935 17511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:19.728087 17511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:19.729063 17511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:19.729383 17511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:19.730497 17511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:19.732110 17511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:19.732215 17511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:19.732338 17511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:19.732414 17511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:19.764518 17511 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:36773
I20260805 22:48:19.764599 17679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:36773 every 8 connection(s)
I20260805 22:48:19.766007 17511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:48:19.774127 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 17511
I20260805 22:48:19.774323 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:48:19.774984 4208 external_mini_cluster.cc:1428] Setting key 5e8a09ea99228f0b88c61d3ed3d27405
I20260805 22:48:19.778183 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34919
--builtin_ntp_servers=127.4.28.20:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260805 22:48:19.779901 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:19.768836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:59301 (local address 127.4.28.62:34919)
0805 22:48:19.769318 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:19.769323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:19.770583 (+ 1260us) server_negotiation.cc:407] Connection header received
0805 22:48:19.772463 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:19.772469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:19.772561 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:19.772664 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:19.774098 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:19.774984 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:19.776868 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:19.777349 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:19.777507 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:19.779027 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:19.779034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:19.779038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:19.779062 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:19.779114 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:19.779131 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:19.779223 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:19.779425 (+ 202us) server_negotiation.cc:299] Negotiation successful
0805 22:48:19.779539 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":234,"threads_started":1}
I20260805 22:48:19.780903 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:19.769362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34919 (local address 127.4.28.1:59301)
0805 22:48:19.770399 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:19.770448 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:19.772132 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:19.772961 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:19.772971 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:19.773071 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:19.773842 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:19.773885 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:19.775181 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:19.775186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:19.776690 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:19.776705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:19.777640 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:19.777649 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:19.777885 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:19.778660 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:48:19.778663 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:19.778710 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:48:19.778803 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:19.779217 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:19.779224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:19.779229 (+ 5us) client_negotiation.cc:769] Sending connection context
0805 22:48:19.779378 (+ 149us) client_negotiation.cc:240] Negotiation successful
0805 22:48:19.779732 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":348,"threads_started":1}
I20260805 22:48:19.782549 17680 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34919
I20260805 22:48:19.782907 17680 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:19.783926 17680 heartbeater.cc:511] Master 127.4.28.62:34919 requested a full tablet report, sending...
I20260805 22:48:19.786501 17429 ts_manager.cc:194] Registered new tserver with Master: b308231dab1f4727b4ba6bb73972631e (127.4.28.1:36773)
I20260805 22:48:19.788090 17429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.1:59301
W20260805 22:48:20.135159 17685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:20.135499 17685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:20.135646 17685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:20.143515 17685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:20.143673 17685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:20.143819 17685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:48:20.153594 17685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:20.155047 17685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:20.157034 17685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:20.171067 17693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:20.171108 17691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:20.171561 17695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:20.171977 17685 server_base.cc:1034] running on GCE node
I20260805 22:48:20.172683 17685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:20.181900 17685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:20.183210 17685 hybrid_clock.cc:648] HybridClock initialized: now 1785970100183127 us; error 105 us; skew 500 ppm
I20260805 22:48:20.183636 17685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:48:20.186486 17685 webserver.cc:460] Webserver started at http://127.4.28.2:45563/ using document root <none> and password file <none>
I20260805 22:48:20.187304 17685 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:20.187428 17685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:20.187836 17685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:20.193954 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "0f5833e28e7448a9be837ffe4d9082f8"
format_stamp: "Formatted at 2026-08-05 22:48:20 on dist-test-slave-7lbf"
server_key: "3760f8595c3a1e0eff8440d3b998999e"
server_key_iv: "5453b7023b6078313af5ecf32a87cfe9"
server_key_version: "encryptionkey@0"
I20260805 22:48:20.194988 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "0f5833e28e7448a9be837ffe4d9082f8"
format_stamp: "Formatted at 2026-08-05 22:48:20 on dist-test-slave-7lbf"
server_key: "3760f8595c3a1e0eff8440d3b998999e"
server_key_iv: "5453b7023b6078313af5ecf32a87cfe9"
server_key_version: "encryptionkey@0"
I20260805 22:48:20.201575 17685 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260805 22:48:20.206023 17705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:20.207653 17685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260805 22:48:20.207831 17685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "0f5833e28e7448a9be837ffe4d9082f8"
format_stamp: "Formatted at 2026-08-05 22:48:20 on dist-test-slave-7lbf"
server_key: "3760f8595c3a1e0eff8440d3b998999e"
server_key_iv: "5453b7023b6078313af5ecf32a87cfe9"
server_key_version: "encryptionkey@0"
I20260805 22:48:20.208041 17685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:20.236235 17685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:20.237221 17685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:20.237566 17685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:20.238683 17685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:20.240370 17685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:20.240464 17685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:20.240582 17685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:20.240649 17685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:20.272078 17685 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:39581
I20260805 22:48:20.272186 17869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:39581 every 8 connection(s)
I20260805 22:48:20.273610 17685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:48:20.283227 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 17685
I20260805 22:48:20.283435 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:48:20.284010 4208 external_mini_cluster.cc:1428] Setting key 1d4ad27376103424d5ae6af993b2b3b4
I20260805 22:48:20.287413 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34919
--builtin_ntp_servers=127.4.28.20:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260805 22:48:20.287575 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:20.276423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:58983 (local address 127.4.28.62:34919)
0805 22:48:20.276606 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:20.276611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:20.278183 (+ 1572us) server_negotiation.cc:407] Connection header received
0805 22:48:20.280079 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:20.280085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:20.280177 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:20.280268 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:20.281446 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.282238 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.284466 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.284928 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.285047 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:20.286694 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:20.286701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:20.286706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:20.286731 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:20.286776 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:20.286793 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:20.286899 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:20.287114 (+ 215us) server_negotiation.cc:299] Negotiation successful
0805 22:48:20.287229 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260805 22:48:20.288622 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:20.276942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34919 (local address 127.4.28.2:58983)
0805 22:48:20.278010 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:20.278052 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:20.279757 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:20.280530 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:20.280539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:20.280648 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:20.281261 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:20.281279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.282404 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.282408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:20.284285 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:20.284304 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.285194 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.285201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:20.285434 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:20.286241 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:48:20.286246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:20.286305 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:48:20.286409 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:20.286892 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:20.286900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:20.286907 (+ 7us) client_negotiation.cc:769] Sending connection context
0805 22:48:20.287072 (+ 165us) client_negotiation.cc:240] Negotiation successful
0805 22:48:20.287421 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":312,"threads_started":1}
I20260805 22:48:20.290232 17870 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34919
I20260805 22:48:20.290573 17870 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:20.291404 17870 heartbeater.cc:511] Master 127.4.28.62:34919 requested a full tablet report, sending...
I20260805 22:48:20.293035 17429 ts_manager.cc:194] Registered new tserver with Master: 0f5833e28e7448a9be837ffe4d9082f8 (127.4.28.2:39581)
I20260805 22:48:20.294113 17429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.2:58983
W20260805 22:48:20.648054 17875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:20.648388 17875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:20.648478 17875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:20.656100 17875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:20.656231 17875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:20.656352 17875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:48:20.666400 17875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:20.667872 17875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:20.669957 17875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:20.684571 17880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:20.684625 17886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:20.684544 17882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:20.685616 17875 server_base.cc:1034] running on GCE node
I20260805 22:48:20.686360 17875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:20.687481 17875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:20.688715 17875 hybrid_clock.cc:648] HybridClock initialized: now 1785970100688643 us; error 64 us; skew 500 ppm
I20260805 22:48:20.689072 17875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:48:20.691635 17875 webserver.cc:460] Webserver started at http://127.4.28.3:35629/ using document root <none> and password file <none>
I20260805 22:48:20.692410 17875 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:20.692528 17875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:20.692873 17875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:20.696092 17875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "1e6ef4740d224173b04ed29190dfce38"
format_stamp: "Formatted at 2026-08-05 22:48:20 on dist-test-slave-7lbf"
server_key: "ab24ac701d4ed40642812bf42f1260fa"
server_key_iv: "978c9ba198d774dd3e1a0e7ed2a068b4"
server_key_version: "encryptionkey@0"
I20260805 22:48:20.696997 17875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "1e6ef4740d224173b04ed29190dfce38"
format_stamp: "Formatted at 2026-08-05 22:48:20 on dist-test-slave-7lbf"
server_key: "ab24ac701d4ed40642812bf42f1260fa"
server_key_iv: "978c9ba198d774dd3e1a0e7ed2a068b4"
server_key_version: "encryptionkey@0"
I20260805 22:48:20.702915 17875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260805 22:48:20.706848 17896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:20.708410 17875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:48:20.708614 17875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "1e6ef4740d224173b04ed29190dfce38"
format_stamp: "Formatted at 2026-08-05 22:48:20 on dist-test-slave-7lbf"
server_key: "ab24ac701d4ed40642812bf42f1260fa"
server_key_iv: "978c9ba198d774dd3e1a0e7ed2a068b4"
server_key_version: "encryptionkey@0"
I20260805 22:48:20.708846 17875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:20.753651 17875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:20.754634 17875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:20.754925 17875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:20.756079 17875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:20.757848 17875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:20.757959 17875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:20.758082 17875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:20.758158 17875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:20.791675 17680 heartbeater.cc:503] Master 127.4.28.62:34919 was elected leader, sending a full tablet report...
I20260805 22:48:20.794360 17875 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:46519
I20260805 22:48:20.794438 18053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:46519 every 8 connection(s)
I20260805 22:48:20.796034 17875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:48:20.799808 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 17875
I20260805 22:48:20.800004 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:48:20.800424 4208 external_mini_cluster.cc:1428] Setting key 810e865a3764fe2c68ab01de05384ad0
I20260805 22:48:20.810163 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:20.798863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:56605 (local address 127.4.28.62:34919)
0805 22:48:20.799308 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:20.799313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:20.800848 (+ 1535us) server_negotiation.cc:407] Connection header received
0805 22:48:20.802943 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:20.802948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:20.803037 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:20.803161 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:20.804532 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.805291 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.807170 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.807601 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.807703 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:20.809294 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:20.809299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:20.809303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:20.809325 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:20.809369 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:20.809382 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:20.809462 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:20.809760 (+ 298us) server_negotiation.cc:299] Negotiation successful
0805 22:48:20.809861 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260805 22:48:20.811326 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:20.799508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34919 (local address 127.4.28.3:56605)
0805 22:48:20.800656 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:20.800713 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:20.802660 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:20.803506 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:20.803518 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:20.803635 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:20.804347 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:20.804368 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.805474 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.805478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:20.806983 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:20.807005 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.807884 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.807891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:20.808131 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:20.808918 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:48:20.808921 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:20.808975 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:48:20.809077 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:20.809485 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:20.809495 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:20.809503 (+ 8us) client_negotiation.cc:769] Sending connection context
0805 22:48:20.809743 (+ 240us) client_negotiation.cc:240] Negotiation successful
0805 22:48:20.810141 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":347,"threads_started":1}
I20260805 22:48:20.812943 18054 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34919
I20260805 22:48:20.813313 18054 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:20.814254 18054 heartbeater.cc:511] Master 127.4.28.62:34919 requested a full tablet report, sending...
I20260805 22:48:20.815903 17429 ts_manager.cc:194] Registered new tserver with Master: 1e6ef4740d224173b04ed29190dfce38 (127.4.28.3:46519)
I20260805 22:48:20.816820 17429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.3:56605
I20260805 22:48:20.829331 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260805 22:48:20.836486 18056 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:20.834665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.4.28.62:34919)
0805 22:48:20.834898 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:20.834906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:20.835333 (+ 427us) server_negotiation.cc:407] Connection header received
0805 22:48:20.835526 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:20.835530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:20.835610 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:20.835702 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:20.836217 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59838: BlockingRecv error: recv got EOF from 127.0.0.1:59838 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260805 22:48:20.837311 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 17511
I20260805 22:48:20.845717 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:20.838406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.4.28.1:36773)
0805 22:48:20.839003 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:20.839011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:20.839039 (+ 28us) server_negotiation.cc:407] Connection header received
0805 22:48:20.839100 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:20.839107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:20.839331 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:20.839460 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:20.839914 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.840954 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.842339 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.842939 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.843062 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:20.844165 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:20.844175 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:20.844191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:20.844807 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:20.844859 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:20.844881 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:20.844990 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:20.845174 (+ 184us) server_negotiation.cc:299] Negotiation successful
0805 22:48:20.845308 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":287,"threads_started":1}
I20260805 22:48:20.847163 17673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:20.930186 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 17511
I20260805 22:48:20.954324 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 17685
I20260805 22:48:20.962634 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:20.955291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35854 (local address 127.4.28.2:39581)
0805 22:48:20.955835 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:20.955843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:20.955874 (+ 31us) server_negotiation.cc:407] Connection header received
0805 22:48:20.955936 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:20.955943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:20.956200 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:20.956334 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:20.956786 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.957886 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.959203 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:20.959820 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:20.959933 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:20.961058 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:20.961067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:20.961083 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:20.961804 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:20.961856 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:20.961879 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:20.961970 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:20.962134 (+ 164us) server_negotiation.cc:299] Negotiation successful
0805 22:48:20.962265 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":247,"threads_started":1}
I20260805 22:48:20.963989 17863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:21.046234 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 17685
I20260805 22:48:21.069259 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 17875
I20260805 22:48:21.077706 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:21.070492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.4.28.3:46519)
0805 22:48:21.071184 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:21.071191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:21.071225 (+ 34us) server_negotiation.cc:407] Connection header received
0805 22:48:21.071288 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:21.071295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:21.071515 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:21.071655 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:21.072122 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:21.073177 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:21.074619 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:21.075194 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:21.075293 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:21.076300 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:21.076309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:21.076325 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:21.076951 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:21.077000 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:21.077033 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:21.077132 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:21.077234 (+ 102us) server_negotiation.cc:299] Negotiation successful
0805 22:48:21.077351 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":340,"threads_started":1}
I20260805 22:48:21.079067 18047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:21.161548 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 17875
I20260805 22:48:21.185451 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 17386
I20260805 22:48:21.186738 17469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:21.293345 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 17386
2026-08-05T22:48:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2687 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-05T22:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:48:21Z Disabled control of system clock
I20260805 22:48:21.383517 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34739
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:34739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260805 22:48:21.739646 18079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:21.739992 18079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:21.740090 18079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:21.747931 18079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:48:21.748071 18079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:21.748111 18079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:21.748169 18079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:48:21.748214 18079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:48:21.758076 18079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:34739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:21.759473 18079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:21.761482 18079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:21.774288 18086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:21.774312 18088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:21.774334 18085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:21.775321 18079 server_base.cc:1034] running on GCE node
I20260805 22:48:21.776204 18079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:21.777846 18079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:21.779062 18079 hybrid_clock.cc:648] HybridClock initialized: now 1785970101779013 us; error 39 us; skew 500 ppm
I20260805 22:48:21.779440 18079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:48:21.781961 18079 webserver.cc:460] Webserver started at http://127.4.28.62:37711/ using document root <none> and password file <none>
I20260805 22:48:21.782742 18079 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:21.782840 18079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:21.783177 18079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:21.786371 18079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "ffd876ead0cf4cd08ee674547c9dea51"
format_stamp: "Formatted at 2026-08-05 22:48:21 on dist-test-slave-7lbf"
server_key: "c0aed3506e9f938362bc7f98e9699e60"
server_key_iv: "3081a8c7eb3328adfd03e9ba464df5ea"
server_key_version: "encryptionkey@0"
I20260805 22:48:21.787236 18079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "ffd876ead0cf4cd08ee674547c9dea51"
format_stamp: "Formatted at 2026-08-05 22:48:21 on dist-test-slave-7lbf"
server_key: "c0aed3506e9f938362bc7f98e9699e60"
server_key_iv: "3081a8c7eb3328adfd03e9ba464df5ea"
server_key_version: "encryptionkey@0"
I20260805 22:48:21.793080 18079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:48:21.797019 18097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:21.798676 18079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:48:21.798857 18079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "ffd876ead0cf4cd08ee674547c9dea51"
format_stamp: "Formatted at 2026-08-05 22:48:21 on dist-test-slave-7lbf"
server_key: "c0aed3506e9f938362bc7f98e9699e60"
server_key_iv: "3081a8c7eb3328adfd03e9ba464df5ea"
server_key_version: "encryptionkey@0"
I20260805 22:48:21.799088 18079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:21.805333 18079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:21.806214 18079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:21.806511 18079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:21.824317 18079 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:34739
I20260805 22:48:21.824337 18165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:34739 every 8 connection(s)
I20260805 22:48:21.825871 18079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:48:21.830082 18166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:48:21.834118 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 18079
I20260805 22:48:21.834321 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:48:21.835018 4208 external_mini_cluster.cc:1428] Setting key ea84f97a44b5b9a9489655b2c343b44a
I20260805 22:48:21.844058 18166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51: Bootstrap starting.
I20260805 22:48:21.846415 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:21.837384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.4.28.62:34739)
0805 22:48:21.838661 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:21.838676 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:21.838738 (+ 62us) server_negotiation.cc:407] Connection header received
0805 22:48:21.841816 (+ 3078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:21.841861 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:21.842533 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:21.843065 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:21.844176 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:21.844184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:21.844203 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:21.844224 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:21.844306 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:21.844333 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:21.844579 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:21.844729 (+ 150us) server_negotiation.cc:299] Negotiation successful
0805 22:48:21.845058 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":391,"threads_started":1}
I20260805 22:48:21.848060 18166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51: Neither blocks nor log segments found. Creating new log.
I20260805 22:48:21.849186 18166 log.cc:824] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51: Log is configured to *not* fsync() on all Append() calls
I20260805 22:48:21.852240 18166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51: No bootstrap required, opened a new log
I20260805 22:48:21.862414 18166 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd876ead0cf4cd08ee674547c9dea51" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34739 } }
I20260805 22:48:21.862864 18166 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:48:21.862987 18166 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd876ead0cf4cd08ee674547c9dea51, State: Initialized, Role: FOLLOWER
I20260805 22:48:21.863675 18166 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd876ead0cf4cd08ee674547c9dea51" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34739 } }
I20260805 22:48:21.863919 18166 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:48:21.864063 18166 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:48:21.864243 18166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:48:21.867059 18166 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd876ead0cf4cd08ee674547c9dea51" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34739 } }
I20260805 22:48:21.867648 18166 leader_election.cc:302] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd876ead0cf4cd08ee674547c9dea51; no voters:
I20260805 22:48:21.868302 18166 leader_election.cc:288] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:48:21.868455 18172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:48:21.869872 18172 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [term 1 LEADER]: Becoming Leader. State: Replica: ffd876ead0cf4cd08ee674547c9dea51, State: Running, Role: LEADER
I20260805 22:48:21.870510 18172 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd876ead0cf4cd08ee674547c9dea51" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34739 } }
I20260805 22:48:21.871089 18166 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:48:21.873613 18175 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd876ead0cf4cd08ee674547c9dea51. Latest consensus state: current_term: 1 leader_uuid: "ffd876ead0cf4cd08ee674547c9dea51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd876ead0cf4cd08ee674547c9dea51" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34739 } } }
I20260805 22:48:21.874058 18175 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [sys.catalog]: This master's current role is: LEADER
I20260805 22:48:21.874606 18174 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd876ead0cf4cd08ee674547c9dea51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd876ead0cf4cd08ee674547c9dea51" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34739 } } }
I20260805 22:48:21.874972 18174 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51 [sys.catalog]: This master's current role is: LEADER
I20260805 22:48:21.876682 18181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:48:21.882352 18181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:48:21.892753 18181 catalog_manager.cc:1347] Generated new cluster ID: ad227101159a455e92572e46f73a04bf
I20260805 22:48:21.892902 18181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:48:21.912341 18181 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:48:21.912820 18181 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:48:21.924263 18181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffd876ead0cf4cd08ee674547c9dea51: Generated new TSK 0
I20260805 22:48:21.925218 18181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:48:21.939424 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34739
--builtin_ntp_servers=127.4.28.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260805 22:48:22.308156 18195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:22.308491 18195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:22.308626 18195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:22.316155 18195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:22.316301 18195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:22.316447 18195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:48:22.326023 18195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:22.327431 18195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:22.329425 18195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:22.343573 18204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:22.343614 18208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:22.343636 18206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:22.344951 18195 server_base.cc:1034] running on GCE node
I20260805 22:48:22.345733 18195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:22.346897 18195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:22.348156 18195 hybrid_clock.cc:648] HybridClock initialized: now 1785970102348083 us; error 53 us; skew 500 ppm
I20260805 22:48:22.348546 18195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:48:22.354401 18195 webserver.cc:460] Webserver started at http://127.4.28.1:33873/ using document root <none> and password file <none>
I20260805 22:48:22.355163 18195 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:22.355265 18195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:22.355577 18195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:22.358881 18195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "d17c42a06eae470eac1bafcb279c963e"
format_stamp: "Formatted at 2026-08-05 22:48:22 on dist-test-slave-7lbf"
server_key: "592b649c874be20fd0ccaedbe47249e2"
server_key_iv: "27855c10c6abbde4b3181002e15dc4e9"
server_key_version: "encryptionkey@0"
I20260805 22:48:22.359758 18195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "d17c42a06eae470eac1bafcb279c963e"
format_stamp: "Formatted at 2026-08-05 22:48:22 on dist-test-slave-7lbf"
server_key: "592b649c874be20fd0ccaedbe47249e2"
server_key_iv: "27855c10c6abbde4b3181002e15dc4e9"
server_key_version: "encryptionkey@0"
I20260805 22:48:22.365576 18195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260805 22:48:22.369602 18219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:22.371244 18195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:48:22.371423 18195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "d17c42a06eae470eac1bafcb279c963e"
format_stamp: "Formatted at 2026-08-05 22:48:22 on dist-test-slave-7lbf"
server_key: "592b649c874be20fd0ccaedbe47249e2"
server_key_iv: "27855c10c6abbde4b3181002e15dc4e9"
server_key_version: "encryptionkey@0"
I20260805 22:48:22.371662 18195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:22.377945 18195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:22.378839 18195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:22.379091 18195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:22.380125 18195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:22.381771 18195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:22.381865 18195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:22.381956 18195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:22.382014 18195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:22.415481 18195 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:32963
I20260805 22:48:22.415539 18393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:32963 every 8 connection(s)
I20260805 22:48:22.416934 18195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:48:22.422204 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 18195
I20260805 22:48:22.422386 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:48:22.423131 4208 external_mini_cluster.cc:1428] Setting key 73014eb6ad61c825fae684f1ce5863c8
I20260805 22:48:22.425928 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:22.419766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:44107 (local address 127.4.28.62:34739)
0805 22:48:22.420245 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:22.420252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:22.421657 (+ 1405us) server_negotiation.cc:407] Connection header received
0805 22:48:22.423517 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:22.423524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:22.423604 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:22.423774 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:22.425178 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:22.425184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:22.425189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:22.425210 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:22.425264 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:22.425280 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:22.425388 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:22.425662 (+ 274us) server_negotiation.cc:299] Negotiation successful
0805 22:48:22.425717 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":218,"threads_started":1}
I20260805 22:48:22.426358 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34739
--builtin_ntp_servers=127.4.28.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260805 22:48:22.427486 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:22.420418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34739 (local address 127.4.28.1:44107)
0805 22:48:22.421423 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:22.421484 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:22.423174 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:22.424055 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:22.424065 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:22.424179 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:22.424851 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:48:22.424853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:22.424900 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:48:22.425005 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:22.425398 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:22.425410 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:22.425419 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 22:48:22.425776 (+ 357us) client_negotiation.cc:240] Negotiation successful
0805 22:48:22.426106 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":293,"threads_started":1}
I20260805 22:48:22.429207 18394 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34739
I20260805 22:48:22.429589 18394 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:22.430276 18394 heartbeater.cc:511] Master 127.4.28.62:34739 requested a full tablet report, sending...
I20260805 22:48:22.432593 18118 ts_manager.cc:194] Registered new tserver with Master: d17c42a06eae470eac1bafcb279c963e (127.4.28.1:32963)
W20260805 22:48:22.787567 18401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:22.788026 18401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:22.788190 18401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:22.799790 18401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:22.799962 18401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:22.800146 18401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:48:22.810140 18401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:22.811740 18401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:22.813678 18401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:22.827256 18409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:22.827471 18414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:22.827736 18408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:22.828552 18401 server_base.cc:1034] running on GCE node
I20260805 22:48:22.829229 18401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:22.830325 18401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:22.831583 18401 hybrid_clock.cc:648] HybridClock initialized: now 1785970102831527 us; error 41 us; skew 500 ppm
I20260805 22:48:22.831964 18401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:48:22.834539 18401 webserver.cc:460] Webserver started at http://127.4.28.2:41273/ using document root <none> and password file <none>
I20260805 22:48:22.835320 18401 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:22.835418 18401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:22.835767 18401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:22.838924 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "568f8520e62b4cd487448de53f25d282"
format_stamp: "Formatted at 2026-08-05 22:48:22 on dist-test-slave-7lbf"
server_key: "558897663047b49d16d390dd61b27ca6"
server_key_iv: "384789846c0815f57cf7d680f4e05113"
server_key_version: "encryptionkey@0"
I20260805 22:48:22.839792 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "568f8520e62b4cd487448de53f25d282"
format_stamp: "Formatted at 2026-08-05 22:48:22 on dist-test-slave-7lbf"
server_key: "558897663047b49d16d390dd61b27ca6"
server_key_iv: "384789846c0815f57cf7d680f4e05113"
server_key_version: "encryptionkey@0"
I20260805 22:48:22.845738 18401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260805 22:48:22.849777 18422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:22.851379 18401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:48:22.851593 18401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "568f8520e62b4cd487448de53f25d282"
format_stamp: "Formatted at 2026-08-05 22:48:22 on dist-test-slave-7lbf"
server_key: "558897663047b49d16d390dd61b27ca6"
server_key_iv: "384789846c0815f57cf7d680f4e05113"
server_key_version: "encryptionkey@0"
I20260805 22:48:22.851835 18401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:22.858278 18401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:22.859268 18401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:22.859562 18401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:22.860654 18401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:22.862345 18401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:22.862452 18401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:22.862577 18401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:22.862648 18401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:22.896003 18401 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:40421
I20260805 22:48:22.896052 18585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:40421 every 8 connection(s)
I20260805 22:48:22.897420 18401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:48:22.898408 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 18401
I20260805 22:48:22.898581 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:48:22.899132 4208 external_mini_cluster.cc:1428] Setting key 7fa2bd4c1a6d9eb73cf9baf74b98568c
I20260805 22:48:22.902699 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34739
--builtin_ntp_servers=127.4.28.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260805 22:48:22.906809 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:22.900391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:48663 (local address 127.4.28.62:34739)
0805 22:48:22.900603 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:22.900610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:22.902368 (+ 1758us) server_negotiation.cc:407] Connection header received
0805 22:48:22.904421 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:22.904426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:22.904518 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:22.904612 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:22.906068 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:22.906074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:22.906077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:22.906096 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:22.906130 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:22.906143 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:22.906271 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:22.906459 (+ 188us) server_negotiation.cc:299] Negotiation successful
0805 22:48:22.906536 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260805 22:48:22.907876 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:22.901000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34739 (local address 127.4.28.2:48663)
0805 22:48:22.902198 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:22.902240 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:22.904112 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:22.904888 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:22.904897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:22.905010 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:22.905715 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:48:22.905718 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:22.905770 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:48:22.905869 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:22.906232 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:22.906243 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:22.906252 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 22:48:22.906535 (+ 283us) client_negotiation.cc:240] Negotiation successful
0805 22:48:22.906829 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":404,"threads_started":1}
I20260805 22:48:22.909456 18586 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34739
I20260805 22:48:22.909830 18586 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:22.910532 18586 heartbeater.cc:511] Master 127.4.28.62:34739 requested a full tablet report, sending...
I20260805 22:48:22.912240 18118 ts_manager.cc:194] Registered new tserver with Master: 568f8520e62b4cd487448de53f25d282 (127.4.28.2:40421)
W20260805 22:48:23.260269 18591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:23.260627 18591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:23.260756 18591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:23.268348 18591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:23.268491 18591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:23.268648 18591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:48:23.278568 18591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:23.280068 18591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:23.282131 18591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:23.296530 18597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:23.296468 18603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:23.296499 18599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:23.298023 18591 server_base.cc:1034] running on GCE node
I20260805 22:48:23.298765 18591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:23.299983 18591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:23.301255 18591 hybrid_clock.cc:648] HybridClock initialized: now 1785970103301190 us; error 58 us; skew 500 ppm
I20260805 22:48:23.301704 18591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:48:23.304666 18591 webserver.cc:460] Webserver started at http://127.4.28.3:38029/ using document root <none> and password file <none>
I20260805 22:48:23.305485 18591 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:23.305665 18591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:23.306039 18591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:23.309376 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "fe499e62c1be45d6abef62904ca3751d"
format_stamp: "Formatted at 2026-08-05 22:48:23 on dist-test-slave-7lbf"
server_key: "1a51fc8ba6aea7cf762b2b86c5d795c9"
server_key_iv: "9143be9448831b7d969d12bb5ca8bca3"
server_key_version: "encryptionkey@0"
I20260805 22:48:23.310408 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "fe499e62c1be45d6abef62904ca3751d"
format_stamp: "Formatted at 2026-08-05 22:48:23 on dist-test-slave-7lbf"
server_key: "1a51fc8ba6aea7cf762b2b86c5d795c9"
server_key_iv: "9143be9448831b7d969d12bb5ca8bca3"
server_key_version: "encryptionkey@0"
I20260805 22:48:23.316521 18591 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260805 22:48:23.320577 18612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:23.322207 18591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:48:23.322397 18591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "fe499e62c1be45d6abef62904ca3751d"
format_stamp: "Formatted at 2026-08-05 22:48:23 on dist-test-slave-7lbf"
server_key: "1a51fc8ba6aea7cf762b2b86c5d795c9"
server_key_iv: "9143be9448831b7d969d12bb5ca8bca3"
server_key_version: "encryptionkey@0"
I20260805 22:48:23.322635 18591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:23.329032 18591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:23.330025 18591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:23.330305 18591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:23.331401 18591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:23.333025 18591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:23.333130 18591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:23.333250 18591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:23.333321 18591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:23.366042 18591 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:46339
I20260805 22:48:23.366083 18778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:46339 every 8 connection(s)
I20260805 22:48:23.367522 18591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:48:23.374292 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 18591
I20260805 22:48:23.374500 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:48:23.375124 4208 external_mini_cluster.cc:1428] Setting key 307bd6a18c848de55c0101aceffdbfe3
I20260805 22:48:23.377717 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:23.370628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:38313 (local address 127.4.28.62:34739)
0805 22:48:23.370880 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:23.370887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:23.372702 (+ 1815us) server_negotiation.cc:407] Connection header received
0805 22:48:23.374824 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:23.374829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:23.374897 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:23.375002 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:23.376848 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:23.376855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:23.376859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:23.376884 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:23.376931 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:23.376950 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:23.377074 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:23.377306 (+ 232us) server_negotiation.cc:299] Negotiation successful
0805 22:48:23.377385 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260805 22:48:23.378890 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:23.371212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34739 (local address 127.4.28.3:38313)
0805 22:48:23.372518 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:23.372563 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:23.374434 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:23.375353 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:23.375368 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:23.375550 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:23.376419 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:48:23.376422 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:23.376500 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:48:23.376632 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:23.377071 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:23.377080 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:23.377088 (+ 8us) client_negotiation.cc:769] Sending connection context
0805 22:48:23.377368 (+ 280us) client_negotiation.cc:240] Negotiation successful
0805 22:48:23.377751 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":416,"threads_started":1}
I20260805 22:48:23.380481 18779 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34739
I20260805 22:48:23.380806 18779 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:23.381419 18779 heartbeater.cc:511] Master 127.4.28.62:34739 requested a full tablet report, sending...
I20260805 22:48:23.383149 18118 ts_manager.cc:194] Registered new tserver with Master: fe499e62c1be45d6abef62904ca3751d (127.4.28.3:46339)
I20260805 22:48:23.392254 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260805 22:48:23.398808 18398 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:48:23.397172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.4.28.62:34739)
0805 22:48:23.397382 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:23.397387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:23.397931 (+ 544us) server_negotiation.cc:407] Connection header received
0805 22:48:23.398100 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:23.398104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:23.398164 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:23.398262 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:23.398578 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43982: BlockingRecv error: recv got EOF from 127.0.0.1:43982 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:48:23.399806 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 18195
I20260805 22:48:23.404577 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:23.400927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.4.28.1:32963)
0805 22:48:23.401501 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:23.401509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:23.401582 (+ 73us) server_negotiation.cc:407] Connection header received
0805 22:48:23.401649 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:23.401656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:23.401878 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:23.402070 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:23.403101 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:23.403111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:23.403207 (+ 96us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:23.403240 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:23.403308 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:23.403337 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:23.403487 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:23.404050 (+ 563us) server_negotiation.cc:299] Negotiation successful
0805 22:48:23.404177 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260805 22:48:23.406570 18387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:23.523093 18394 heartbeater.cc:503] Master 127.4.28.62:34739 was elected leader, sending a full tablet report...
I20260805 22:48:23.524809 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 18195
W20260805 22:48:23.556635 18098 connection.cc:535] server connection from 127.4.28.1:44107 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260805 22:48:23.556950 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 18401
I20260805 22:48:23.561334 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:23.558133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.4.28.2:40421)
0805 22:48:23.558884 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:23.558892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:23.558923 (+ 31us) server_negotiation.cc:407] Connection header received
0805 22:48:23.558992 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:23.558999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:23.559250 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:23.559443 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:23.560525 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:23.560533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:23.560548 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:23.560563 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:23.560614 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:23.560630 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:23.560766 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:23.560916 (+ 150us) server_negotiation.cc:299] Negotiation successful
0805 22:48:23.561029 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":419,"threads_started":1}
I20260805 22:48:23.562852 18581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:23.644153 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 18401
I20260805 22:48:23.667582 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 18591
I20260805 22:48:23.671626 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:23.668583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.4.28.3:46339)
0805 22:48:23.669239 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:23.669247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:23.669282 (+ 35us) server_negotiation.cc:407] Connection header received
0805 22:48:23.669352 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:23.669359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:23.669666 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:23.669814 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:23.670886 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:23.670893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:23.670909 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:48:23.670922 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:23.670965 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:48:23.670983 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:23.671142 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:23.671203 (+ 61us) server_negotiation.cc:299] Negotiation successful
0805 22:48:23.671300 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":308,"threads_started":1}
I20260805 22:48:23.673143 18772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:23.756170 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 18591
I20260805 22:48:23.782521 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 18079
I20260805 22:48:23.783886 18161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:23.889657 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 18079
2026-08-05T22:48:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2598 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:48:24 dist-test-slave-7lbf krb5kdc[18803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:48:24 dist-test-slave-7lbf krb5kdc[18803](info): set up 2 sockets
Aug 05 22:48:24 dist-test-slave-7lbf krb5kdc[18803](info): commencing operation
krb5kdc: starting...
W20260805 22:48:25.125701 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:48:25 dist-test-slave-7lbf krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:48:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:48:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:48:25.434501 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:41349
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:48:25.790063 18821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:25.790396 18821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:25.790519 18821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:25.798276 18821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:48:25.798414 18821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:25.798470 18821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:25.798554 18821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:48:25.798617 18821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:48:25.809191 18821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:25.810745 18821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:25.812232 18821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:25.824903 18827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:25.824896 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:25.824941 18830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:25.825918 18821 server_base.cc:1034] running on GCE node
I20260805 22:48:25.826792 18821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:25.828361 18821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:25.829614 18821 hybrid_clock.cc:648] HybridClock initialized: now 1785970105829550 us; error 48 us; skew 500 ppm
I20260805 22:48:25.829988 18821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:48:25 dist-test-slave-7lbf krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:48:25.833374 18821 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:48:25.835047 18821 webserver.cc:460] Webserver started at http://127.4.28.62:41717/ using document root <none> and password file <none>
I20260805 22:48:25.835767 18821 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:25.835896 18821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:25.836213 18821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:25.839489 18821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "9bcd0c3c5fc147309ffefc7bb59ea152"
format_stamp: "Formatted at 2026-08-05 22:48:25 on dist-test-slave-7lbf"
server_key: "ec742166c0e01356b5d1523c4d02c56d"
server_key_iv: "0ce8a94c85bc3ac2af687164c70553d1"
server_key_version: "encryptionkey@0"
I20260805 22:48:25.840358 18821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "9bcd0c3c5fc147309ffefc7bb59ea152"
format_stamp: "Formatted at 2026-08-05 22:48:25 on dist-test-slave-7lbf"
server_key: "ec742166c0e01356b5d1523c4d02c56d"
server_key_iv: "0ce8a94c85bc3ac2af687164c70553d1"
server_key_version: "encryptionkey@0"
I20260805 22:48:25.846325 18821 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260805 22:48:25.850260 18839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:25.851866 18821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:48:25.852075 18821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "9bcd0c3c5fc147309ffefc7bb59ea152"
format_stamp: "Formatted at 2026-08-05 22:48:25 on dist-test-slave-7lbf"
server_key: "ec742166c0e01356b5d1523c4d02c56d"
server_key_iv: "0ce8a94c85bc3ac2af687164c70553d1"
server_key_version: "encryptionkey@0"
I20260805 22:48:25.852283 18821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:25.892603 18821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:25.897137 18821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:25.897434 18821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:25.916795 18821 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:41349
I20260805 22:48:25.916803 18912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:41349 every 8 connection(s)
I20260805 22:48:25.918386 18821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:48:25.922416 18913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:48:25.925472 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 18821
I20260805 22:48:25.925729 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:48:25.926559 4208 external_mini_cluster.cc:1428] Setting key c65e0b4ceaca397c9ffb78166728ef47
I20260805 22:48:25.936045 18913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152: Bootstrap starting.
I20260805 22:48:25.939718 18913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152: Neither blocks nor log segments found. Creating new log.
I20260805 22:48:25.940884 18913 log.cc:824] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152: Log is configured to *not* fsync() on all Append() calls
Aug 05 22:48:25 dist-test-slave-7lbf krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:48:25.944401 18913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152: No bootstrap required, opened a new log
I20260805 22:48:25.950958 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:25.929099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.4.28.62:41349)
0805 22:48:25.929912 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:25.929925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:25.929972 (+ 47us) server_negotiation.cc:407] Connection header received
0805 22:48:25.931768 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:25.931795 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:25.932295 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:25.932673 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:25.934724 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:25.936122 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:25.937549 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:25.938343 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:25.938526 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:25.943200 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:25.943210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:25.943227 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:25.943277 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:25.946484 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:25.947244 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:25.947252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:25.947260 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:25.947454 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:25.947864 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:25.947868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:25.947870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:25.948468 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:25.948801 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:25.949298 (+ 497us) server_negotiation.cc:299] Negotiation successful
0805 22:48:25.949810 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":369,"threads_started":1}
I20260805 22:48:25.952889 18913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcd0c3c5fc147309ffefc7bb59ea152" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41349 } }
I20260805 22:48:25.953398 18913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:48:25.953613 18913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcd0c3c5fc147309ffefc7bb59ea152, State: Initialized, Role: FOLLOWER
I20260805 22:48:25.954298 18913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcd0c3c5fc147309ffefc7bb59ea152" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41349 } }
I20260805 22:48:25.954602 18913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:48:25.954722 18913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:48:25.954856 18913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:48:25.957690 18913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcd0c3c5fc147309ffefc7bb59ea152" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41349 } }
I20260805 22:48:25.958232 18913 leader_election.cc:302] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bcd0c3c5fc147309ffefc7bb59ea152; no voters:
I20260805 22:48:25.958825 18913 leader_election.cc:288] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:48:25.958971 18921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:48:25.959606 18921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [term 1 LEADER]: Becoming Leader. State: Replica: 9bcd0c3c5fc147309ffefc7bb59ea152, State: Running, Role: LEADER
I20260805 22:48:25.960180 18921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcd0c3c5fc147309ffefc7bb59ea152" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41349 } }
I20260805 22:48:25.960961 18913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:48:25.963093 18924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcd0c3c5fc147309ffefc7bb59ea152. Latest consensus state: current_term: 1 leader_uuid: "9bcd0c3c5fc147309ffefc7bb59ea152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcd0c3c5fc147309ffefc7bb59ea152" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41349 } } }
I20260805 22:48:25.963558 18924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [sys.catalog]: This master's current role is: LEADER
I20260805 22:48:25.963449 18923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bcd0c3c5fc147309ffefc7bb59ea152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcd0c3c5fc147309ffefc7bb59ea152" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41349 } } }
I20260805 22:48:25.963737 18923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152 [sys.catalog]: This master's current role is: LEADER
I20260805 22:48:25.964404 18929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:48:25.970265 18929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:48:25.985739 18929 catalog_manager.cc:1347] Generated new cluster ID: dc7e821ed92d41048b2fce354c01723d
I20260805 22:48:25.985877 18929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:48:26.029774 18929 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:48:26.031872 18929 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:48:26.042501 18929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152: Generated new TSK 0
I20260805 22:48:26.043718 18929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:48:26.131779 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41349
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:48:26.500569 18955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:26.500907 18955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:26.501060 18955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:26.508670 18955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:26.508857 18955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:26.509025 18955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:48:26.519364 18955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:26.520776 18955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:26.522217 18955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:26.536044 18964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:26.536028 18967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:26.536307 18970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:26.537197 18955 server_base.cc:1034] running on GCE node
I20260805 22:48:26.537972 18955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:26.538992 18955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:26.540237 18955 hybrid_clock.cc:648] HybridClock initialized: now 1785970106540182 us; error 38 us; skew 500 ppm
I20260805 22:48:26.540608 18955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:48:26 dist-test-slave-7lbf krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:48:26.544013 18955 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:48:26.545650 18955 webserver.cc:460] Webserver started at http://127.4.28.1:45943/ using document root <none> and password file <none>
I20260805 22:48:26.546379 18955 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:26.546504 18955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:26.546820 18955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:26.550042 18955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "73fcd475988c406d9adc1cb03a914246"
format_stamp: "Formatted at 2026-08-05 22:48:26 on dist-test-slave-7lbf"
server_key: "2bca8e55744bc4fd089fe9ae79fe89cf"
server_key_iv: "f7c5b237f36f99182fed70b20a6e26f1"
server_key_version: "encryptionkey@0"
I20260805 22:48:26.550926 18955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "73fcd475988c406d9adc1cb03a914246"
format_stamp: "Formatted at 2026-08-05 22:48:26 on dist-test-slave-7lbf"
server_key: "2bca8e55744bc4fd089fe9ae79fe89cf"
server_key_iv: "f7c5b237f36f99182fed70b20a6e26f1"
server_key_version: "encryptionkey@0"
I20260805 22:48:26.556710 18955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:48:26.560719 18977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:26.562414 18955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:48:26.562600 18955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "73fcd475988c406d9adc1cb03a914246"
format_stamp: "Formatted at 2026-08-05 22:48:26 on dist-test-slave-7lbf"
server_key: "2bca8e55744bc4fd089fe9ae79fe89cf"
server_key_iv: "f7c5b237f36f99182fed70b20a6e26f1"
server_key_version: "encryptionkey@0"
I20260805 22:48:26.562817 18955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:26.603487 18955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:26.608184 18955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:26.608512 18955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:26.609637 18955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:26.611204 18955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:26.611292 18955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:26.611431 18955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:26.611487 18955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:26.644186 18955 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:42475
I20260805 22:48:26.644265 19151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:42475 every 8 connection(s)
I20260805 22:48:26.645627 18955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:48:26.646149 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 18955
I20260805 22:48:26.646350 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:48:26.647219 4208 external_mini_cluster.cc:1428] Setting key 01e0a47f5e61eed722b5c38453d4a3e5
Aug 05 22:48:26 dist-test-slave-7lbf krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970106, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:48:26.669577 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:26.648528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:42501 (local address 127.4.28.62:41349)
0805 22:48:26.649058 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:26.649064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:26.650431 (+ 1367us) server_negotiation.cc:407] Connection header received
0805 22:48:26.652497 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:26.652503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:26.652601 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:26.652733 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:26.654743 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:26.655767 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:26.657833 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:26.658403 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:26.658555 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:26.662672 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:26.662681 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:26.662686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:26.662737 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:26.665731 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:26.666715 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:26.666723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:26.666727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:26.666956 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:26.667535 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:26.667539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:26.667542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:26.667883 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:26.668033 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:26.668896 (+ 863us) server_negotiation.cc:299] Negotiation successful
0805 22:48:26.669170 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
I20260805 22:48:26.670734 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:26.649331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:41349 (local address 127.4.28.1:42501)
0805 22:48:26.650244 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:26.650292 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:26.652167 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:26.653032 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:26.653043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:26.653737 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:26.654488 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:26.654510 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:26.656059 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:26.656065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:26.657651 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:26.657668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:26.658653 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:26.658659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:26.658943 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:26.659662 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:26.659689 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:26.662353 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:26.665926 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:26.665935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:26.665953 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:26.666519 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:26.667089 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:26.667093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:26.667095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:26.667380 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:26.668052 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:26.668062 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:26.668458 (+ 396us) client_negotiation.cc:769] Sending connection context
0805 22:48:26.668737 (+ 279us) client_negotiation.cc:240] Negotiation successful
0805 22:48:26.669407 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":401,"threads_started":1}
I20260805 22:48:26.672544 19152 heartbeater.cc:347] Connected to a master server at 127.4.28.62:41349
I20260805 22:48:26.672868 19152 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:26.673700 19152 heartbeater.cc:511] Master 127.4.28.62:41349 requested a full tablet report, sending...
I20260805 22:48:26.675979 18862 ts_manager.cc:194] Registered new tserver with Master: 73fcd475988c406d9adc1cb03a914246 (127.4.28.1:42475)
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
I20260805 22:48:26.677563 18862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:42501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:48:26.714501 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41349
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:48:27.074077 19161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:27.074396 19161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:27.074524 19161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:27.082022 19161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:27.082154 19161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:27.082314 19161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:48:27.092478 19161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:27.093969 19161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:27.095407 19161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:27.109725 19166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:27.109697 19171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:27.109766 19168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:27.110425 19161 server_base.cc:1034] running on GCE node
I20260805 22:48:27.111127 19161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:27.112143 19161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:27.113397 19161 hybrid_clock.cc:648] HybridClock initialized: now 1785970107113320 us; error 62 us; skew 500 ppm
I20260805 22:48:27.113798 19161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:48:27 dist-test-slave-7lbf krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:48:27.117451 19161 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:48:27.119267 19161 webserver.cc:460] Webserver started at http://127.4.28.2:33985/ using document root <none> and password file <none>
I20260805 22:48:27.120034 19161 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:27.120151 19161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:27.120435 19161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:27.123720 19161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "c32d6164b69140318917f2d1f39afa5c"
format_stamp: "Formatted at 2026-08-05 22:48:27 on dist-test-slave-7lbf"
server_key: "733091a25777cb8957ea8e0b1a9293b8"
server_key_iv: "3702d02e5129c60be83c79dd7a5c77ec"
server_key_version: "encryptionkey@0"
I20260805 22:48:27.124604 19161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "c32d6164b69140318917f2d1f39afa5c"
format_stamp: "Formatted at 2026-08-05 22:48:27 on dist-test-slave-7lbf"
server_key: "733091a25777cb8957ea8e0b1a9293b8"
server_key_iv: "3702d02e5129c60be83c79dd7a5c77ec"
server_key_version: "encryptionkey@0"
I20260805 22:48:27.130422 19161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260805 22:48:27.134195 19182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:27.135694 19161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:48:27.135848 19161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "c32d6164b69140318917f2d1f39afa5c"
format_stamp: "Formatted at 2026-08-05 22:48:27 on dist-test-slave-7lbf"
server_key: "733091a25777cb8957ea8e0b1a9293b8"
server_key_iv: "3702d02e5129c60be83c79dd7a5c77ec"
server_key_version: "encryptionkey@0"
I20260805 22:48:27.136037 19161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:27.154582 19161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:27.159502 19161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:27.159807 19161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:27.160861 19161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:27.162591 19161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:27.162674 19161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:27.162786 19161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:27.162839 19161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:27.195281 19161 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:37713
I20260805 22:48:27.195322 19343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:37713 every 8 connection(s)
I20260805 22:48:27.196702 19161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:48:27.205404 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 19161
I20260805 22:48:27.205647 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:48:27.206157 4208 external_mini_cluster.cc:1428] Setting key 591abb887d5de1a37dc0a42130b8b992
Aug 05 22:48:27 dist-test-slave-7lbf krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:48:27.220549 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.199565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:58813 (local address 127.4.28.62:41349)
0805 22:48:27.200048 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:27.200053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:27.201080 (+ 1027us) server_negotiation.cc:407] Connection header received
0805 22:48:27.203197 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:27.203204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:27.203301 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:27.203404 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:27.205593 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.206547 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.208588 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.209082 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.209186 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.213885 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:27.213891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:27.213895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:27.213939 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:27.216649 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:27.217765 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:27.217771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:27.217773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:27.218004 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:27.218626 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:27.218631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:27.218633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:27.218945 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:27.219073 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:27.220046 (+ 973us) server_negotiation.cc:299] Negotiation successful
0805 22:48:27.220253 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260805 22:48:27.221791 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.200034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:41349 (local address 127.4.28.2:58813)
0805 22:48:27.200860 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:27.200915 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:27.202831 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:27.203729 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:27.203740 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:27.204348 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:27.205209 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.205251 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.206753 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.206757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.208393 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.208413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.209353 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.209361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.209699 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.210456 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:27.210497 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:27.213485 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:27.216869 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:27.216881 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:27.216903 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:27.217614 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:27.218141 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:27.218146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:27.218149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:27.218440 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:27.219084 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:27.219093 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:27.219595 (+ 502us) client_negotiation.cc:769] Sending connection context
0805 22:48:27.219913 (+ 318us) client_negotiation.cc:240] Negotiation successful
0805 22:48:27.220526 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":326,"threads_started":1}
I20260805 22:48:27.223483 19344 heartbeater.cc:347] Connected to a master server at 127.4.28.62:41349
I20260805 22:48:27.223822 19344 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:27.224643 19344 heartbeater.cc:511] Master 127.4.28.62:41349 requested a full tablet report, sending...
I20260805 22:48:27.226392 18862 ts_manager.cc:194] Registered new tserver with Master: c32d6164b69140318917f2d1f39afa5c (127.4.28.2:37713)
I20260805 22:48:27.227310 18862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:58813
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:48:27.277556 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41349
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:48:27.635851 19355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:48:27.636183 19355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:48:27.636307 19355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:48:27.644466 19355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:48:27.644620 19355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:48:27.644793 19355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:48:27.655561 19355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:48:27.657150 19355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:48:27.658738 19355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:48:27.673139 19363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:27.673106 19372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:48:27.673170 19369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:48:27.674058 19355 server_base.cc:1034] running on GCE node
I20260805 22:48:27.674738 19355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:48:27.675835 19355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:48:27.677103 19355 hybrid_clock.cc:648] HybridClock initialized: now 1785970107677027 us; error 59 us; skew 500 ppm
I20260805 22:48:27.677498 19355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:48:27.681057 19152 heartbeater.cc:503] Master 127.4.28.62:41349 was elected leader, sending a full tablet report...
Aug 05 22:48:27 dist-test-slave-7lbf krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:48:27.685065 19355 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:48:27.686702 19355 webserver.cc:460] Webserver started at http://127.4.28.3:44185/ using document root <none> and password file <none>
I20260805 22:48:27.687446 19355 fs_manager.cc:359] Metadata directory not provided
I20260805 22:48:27.687575 19355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:48:27.687901 19355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:48:27.691089 19355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "d8b0e58bfc2d45d29d83658e566d5bb4"
format_stamp: "Formatted at 2026-08-05 22:48:27 on dist-test-slave-7lbf"
server_key: "5b495c3a5dcc9eb73c4b69be5016e28e"
server_key_iv: "275ac6e19ec5fde034e281c57a88ccfb"
server_key_version: "encryptionkey@0"
I20260805 22:48:27.691907 19355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "d8b0e58bfc2d45d29d83658e566d5bb4"
format_stamp: "Formatted at 2026-08-05 22:48:27 on dist-test-slave-7lbf"
server_key: "5b495c3a5dcc9eb73c4b69be5016e28e"
server_key_iv: "275ac6e19ec5fde034e281c57a88ccfb"
server_key_version: "encryptionkey@0"
I20260805 22:48:27.697538 19355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260805 22:48:27.701362 19380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:27.702973 19355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:48:27.703148 19355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "d8b0e58bfc2d45d29d83658e566d5bb4"
format_stamp: "Formatted at 2026-08-05 22:48:27 on dist-test-slave-7lbf"
server_key: "5b495c3a5dcc9eb73c4b69be5016e28e"
server_key_iv: "275ac6e19ec5fde034e281c57a88ccfb"
server_key_version: "encryptionkey@0"
I20260805 22:48:27.703356 19355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:48:27.721555 19355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:48:27.726676 19355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:48:27.727008 19355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:48:27.728142 19355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:48:27.729923 19355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:48:27.730008 19355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:27.730146 19355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:48:27.730196 19355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:48:27.760731 19355 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:42383
I20260805 22:48:27.760772 19547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:42383 every 8 connection(s)
I20260805 22:48:27.762265 19355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:48:27.769321 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 19355
I20260805 22:48:27.769503 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:48:27.770008 4208 external_mini_cluster.cc:1428] Setting key 7163761077e6b49d166143947a3cc8a4
Aug 05 22:48:27 dist-test-slave-7lbf krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:48:27.784710 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.765176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:53057 (local address 127.4.28.62:41349)
0805 22:48:27.765738 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:27.765743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:27.766845 (+ 1102us) server_negotiation.cc:407] Connection header received
0805 22:48:27.768888 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:27.768895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:27.768998 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:27.769132 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:27.770936 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.771798 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.773921 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.774334 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.774441 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.778509 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:27.778523 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:27.778527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:27.778571 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:27.781130 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:27.782113 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:27.782120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:27.782124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:27.782291 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:27.782782 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:27.782786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:27.782789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:27.783070 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:27.783182 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:27.784123 (+ 941us) server_negotiation.cc:299] Negotiation successful
0805 22:48:27.784373 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":234,"threads_started":1}
I20260805 22:48:27.786057 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.765814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:41349 (local address 127.4.28.3:53057)
0805 22:48:27.766667 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:27.766718 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:27.768521 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:27.769393 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:27.769402 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:27.770048 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:48:27.770740 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.770768 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.771999 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.772004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.773771 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.773796 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.774571 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.774577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.774860 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.775548 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:48:27.775578 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:48:27.778202 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:48:27.781333 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:27.781343 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:27.781360 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:27.781950 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:27.782431 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:48:27.782436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:48:27.782439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:48:27.782677 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:48:27.783193 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:48:27.783219 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:48:27.783636 (+ 417us) client_negotiation.cc:769] Sending connection context
0805 22:48:27.783996 (+ 360us) client_negotiation.cc:240] Negotiation successful
0805 22:48:27.784548 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":337,"threads_started":1}
I20260805 22:48:27.787407 19548 heartbeater.cc:347] Connected to a master server at 127.4.28.62:41349
I20260805 22:48:27.787739 19548 heartbeater.cc:464] Registering TS with master...
I20260805 22:48:27.788532 19548 heartbeater.cc:511] Master 127.4.28.62:41349 requested a full tablet report, sending...
I20260805 22:48:27.790130 18862 ts_manager.cc:194] Registered new tserver with Master: d8b0e58bfc2d45d29d83658e566d5bb4 (127.4.28.3:42383)
I20260805 22:48:27.790967 18862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:53057
I20260805 22:48:27.799135 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:48:27.816828 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.803893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.4.28.62:41349)
0805 22:48:27.804100 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:27.804106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:27.804564 (+ 458us) server_negotiation.cc:407] Connection header received
0805 22:48:27.804750 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:27.804755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:27.804828 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:27.804965 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:27.806477 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.807387 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.808745 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.809200 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.809308 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.811636 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:27.811641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:27.811645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:27.811697 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:27.814372 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:27.815020 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:27.815024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:27.815026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:27.815159 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:27.815504 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:27.815509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:27.815511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:27.815868 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:27.816104 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:27.816384 (+ 280us) server_negotiation.cc:299] Negotiation successful
0805 22:48:27.816586 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:48:27.821128 18862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:48:27.828706 18862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:48:27.855603 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.842854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:37713 (local address 127.0.0.1:45842)
0805 22:48:27.844229 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:27.844291 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:27.844503 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:27.845706 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:27.845715 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:27.845764 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:48:27.846269 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.846285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.848687 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.848695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.851511 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.851538 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.854308 (+ 2770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.854317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.854624 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.854697 (+ 73us) client_negotiation.cc:769] Sending connection context
0805 22:48:27.854890 (+ 193us) client_negotiation.cc:240] Negotiation successful
0805 22:48:27.855136 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":900,"threads_started":1}
I20260805 22:48:27.857363 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.843081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.4.28.2:37713)
0805 22:48:27.843745 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:27.843756 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:27.844431 (+ 675us) server_negotiation.cc:407] Connection header received
0805 22:48:27.844829 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:27.844840 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:27.845143 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:27.845352 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:48:27.846498 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.848443 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.851803 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.853649 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.853819 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.854097 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:27.856778 (+ 2681us) server_negotiation.cc:299] Negotiation successful
0805 22:48:27.856991 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":319,"threads_started":1}
I20260805 22:48:27.863444 19254 tablet_service.cc:1467] Processing CreateTablet for tablet e1189f8a3f864b908926d123e373120e (DEFAULT_TABLE table=test-table [id=6ed2624a4a5c4e61ab8f8af8ee956f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:48:27.864959 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.843329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.0.0.1:37570)
0805 22:48:27.856359 (+ 13030us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:27.856383 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:27.856543 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:27.857998 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:27.858003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:27.858033 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:48:27.858391 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.858402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.859946 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.859951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.862441 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.862458 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.864082 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.864088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.864282 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.864312 (+ 30us) client_negotiation.cc:769] Sending connection context
0805 22:48:27.864403 (+ 91us) client_negotiation.cc:240] Negotiation successful
0805 22:48:27.864556 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12829,"thread_start_us":12641,"threads_started":1}
I20260805 22:48:27.864996 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.843379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.4.28.1:42475)
0805 22:48:27.850105 (+ 6726us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:27.850116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:27.856711 (+ 6595us) server_negotiation.cc:407] Connection header received
0805 22:48:27.856856 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:27.856866 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:27.857752 (+ 886us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:27.857983 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:48:27.858593 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.859787 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.862752 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.863834 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.863972 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.864230 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:27.864476 (+ 246us) server_negotiation.cc:299] Negotiation successful
0805 22:48:27.864671 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6523,"thread_start_us":6336,"threads_started":1}
I20260805 22:48:27.865708 19254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1189f8a3f864b908926d123e373120e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:48:27.868261 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.844552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:42383 (local address 127.0.0.1:38254)
0805 22:48:27.856486 (+ 11934us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:27.856501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:27.856570 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:27.860660 (+ 4090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:27.860666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:27.860697 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:48:27.860994 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.861009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.862969 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.862975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.864722 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.864739 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.867483 (+ 2744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.867490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.867679 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.867714 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:48:27.867807 (+ 93us) client_negotiation.cc:240] Negotiation successful
0805 22:48:27.867960 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11834,"thread_start_us":13214,"threads_started":1}
I20260805 22:48:27.869216 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.844964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.4.28.3:42383)
0805 22:48:27.860062 (+ 15098us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:27.860071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:27.860109 (+ 38us) server_negotiation.cc:407] Connection header received
0805 22:48:27.860217 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:27.860225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:27.860456 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:27.860644 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:48:27.861173 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.862785 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.864938 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.866979 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.867114 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.867366 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:27.868690 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0805 22:48:27.868867 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14882,"thread_start_us":15091,"threads_started":1}
I20260805 22:48:27.869459 19043 tablet_service.cc:1467] Processing CreateTablet for tablet e1189f8a3f864b908926d123e373120e (DEFAULT_TABLE table=test-table [id=6ed2624a4a5c4e61ab8f8af8ee956f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:48:27.871068 19043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1189f8a3f864b908926d123e373120e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:48:27.873477 19454 tablet_service.cc:1467] Processing CreateTablet for tablet e1189f8a3f864b908926d123e373120e (DEFAULT_TABLE table=test-table [id=6ed2624a4a5c4e61ab8f8af8ee956f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:48:27.875051 19454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1189f8a3f864b908926d123e373120e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:48:27.879930 19581 tablet_bootstrap.cc:492] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: Bootstrap starting.
I20260805 22:48:27.884580 19581 tablet_bootstrap.cc:654] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: Neither blocks nor log segments found. Creating new log.
I20260805 22:48:27.885901 19581 log.cc:824] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: Log is configured to *not* fsync() on all Append() calls
I20260805 22:48:27.886951 19582 tablet_bootstrap.cc:492] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: Bootstrap starting.
I20260805 22:48:27.891242 19585 tablet_bootstrap.cc:492] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: Bootstrap starting.
I20260805 22:48:27.891454 19581 tablet_bootstrap.cc:492] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: No bootstrap required, opened a new log
I20260805 22:48:27.891607 19582 tablet_bootstrap.cc:654] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: Neither blocks nor log segments found. Creating new log.
I20260805 22:48:27.891806 19581 ts_tablet_manager.cc:1397] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260805 22:48:27.893302 19582 log.cc:824] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: Log is configured to *not* fsync() on all Append() calls
I20260805 22:48:27.895869 19585 tablet_bootstrap.cc:654] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: Neither blocks nor log segments found. Creating new log.
I20260805 22:48:27.897364 19582 tablet_bootstrap.cc:492] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: No bootstrap required, opened a new log
I20260805 22:48:27.897382 19585 log.cc:824] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: Log is configured to *not* fsync() on all Append() calls
I20260805 22:48:27.897761 19582 ts_tablet_manager.cc:1397] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260805 22:48:27.900508 19581 raft_consensus.cc:348] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } }
I20260805 22:48:27.900967 19581 raft_consensus.cc:374] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:48:27.901074 19581 raft_consensus.cc:729] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32d6164b69140318917f2d1f39afa5c, State: Initialized, Role: FOLLOWER
I20260805 22:48:27.901785 19581 consensus_queue.cc:260] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } }
I20260805 22:48:27.903033 19344 heartbeater.cc:503] Master 127.4.28.62:41349 was elected leader, sending a full tablet report...
I20260805 22:48:27.903657 19581 ts_tablet_manager.cc:1428] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260805 22:48:27.907811 19585 tablet_bootstrap.cc:492] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: No bootstrap required, opened a new log
I20260805 22:48:27.908152 19585 ts_tablet_manager.cc:1397] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260805 22:48:27.907991 19582 raft_consensus.cc:348] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } }
I20260805 22:48:27.908483 19582 raft_consensus.cc:374] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:48:27.908599 19582 raft_consensus.cc:729] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fcd475988c406d9adc1cb03a914246, State: Initialized, Role: FOLLOWER
I20260805 22:48:27.909281 19582 consensus_queue.cc:260] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } }
I20260805 22:48:27.912081 19582 ts_tablet_manager.cc:1428] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260805 22:48:27.916977 19585 raft_consensus.cc:348] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } }
I20260805 22:48:27.917397 19585 raft_consensus.cc:374] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:48:27.917496 19585 raft_consensus.cc:729] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b0e58bfc2d45d29d83658e566d5bb4, State: Initialized, Role: FOLLOWER
I20260805 22:48:27.918123 19585 consensus_queue.cc:260] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } }
I20260805 22:48:27.919226 19548 heartbeater.cc:503] Master 127.4.28.62:41349 was elected leader, sending a full tablet report...
I20260805 22:48:27.919871 19585 ts_tablet_manager.cc:1428] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
W20260805 22:48:27.948876 19349 tablet.cc:2369] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:48:27.978790 19589 raft_consensus.cc:482] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:48:27.979128 19589 raft_consensus.cc:504] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } }
I20260805 22:48:27.981065 19589 leader_election.cc:288] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73fcd475988c406d9adc1cb03a914246 (127.4.28.1:42475), d8b0e58bfc2d45d29d83658e566d5bb4 (127.4.28.3:42383)
I20260805 22:48:27.988620 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.981681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:60135 (local address 127.4.28.1:42475)
0805 22:48:27.981922 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:27.981929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:27.982701 (+ 772us) server_negotiation.cc:407] Connection header received
0805 22:48:27.983021 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:27.983027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:27.983114 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:27.983211 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:48:27.984056 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.984934 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.986901 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.987681 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.987805 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.987926 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:27.988239 (+ 313us) server_negotiation.cc:299] Negotiation successful
0805 22:48:27.988364 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260805 22:48:27.988667 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.981767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:41477 (local address 127.4.28.3:42383)
0805 22:48:27.981974 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:27.981981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:27.983072 (+ 1091us) server_negotiation.cc:407] Connection header received
0805 22:48:27.983210 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:27.983215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:27.983293 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:27.983394 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:48:27.984160 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.985027 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.986904 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.987704 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.987823 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.987932 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:27.988271 (+ 339us) server_negotiation.cc:299] Negotiation successful
0805 22:48:27.988403 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:48:27.988803 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.981551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.4.28.2:60135)
0805 22:48:27.982546 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:27.982574 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:27.982804 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:27.983299 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:27.983305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:27.983332 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:48:27.983862 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.983879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.985081 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.985086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.986723 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.986738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.987940 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.987950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.988077 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.988102 (+ 25us) client_negotiation.cc:769] Sending connection context
0805 22:48:27.988190 (+ 88us) client_negotiation.cc:240] Negotiation successful
0805 22:48:27.988341 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":21,"spinlock_wait_cycles":18432,"thread_start_us":1032,"threads_started":1}
I20260805 22:48:27.989301 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:27.981638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:42383 (local address 127.4.28.2:41477)
0805 22:48:27.982959 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:27.982973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 22:48:27.983088 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:48:27.983454 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:48:27.983459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:48:27.983510 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:48:27.983989 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.984000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.985177 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.985182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.986738 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:48:27.986752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:48:27.987968 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:48:27.987974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:48:27.988112 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:27.988136 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 22:48:27.988217 (+ 81us) client_negotiation.cc:240] Negotiation successful
0805 22:48:27.988346 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":23,"spinlock_wait_cycles":1920,"thread_start_us":522,"threads_started":1}
I20260805 22:48:27.989920 19077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1189f8a3f864b908926d123e373120e" candidate_uuid: "c32d6164b69140318917f2d1f39afa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73fcd475988c406d9adc1cb03a914246" is_pre_election: true
I20260805 22:48:27.990384 19484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1189f8a3f864b908926d123e373120e" candidate_uuid: "c32d6164b69140318917f2d1f39afa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" is_pre_election: true
I20260805 22:48:27.990607 19077 raft_consensus.cc:2454] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c32d6164b69140318917f2d1f39afa5c in term 0.
I20260805 22:48:27.990974 19484 raft_consensus.cc:2454] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c32d6164b69140318917f2d1f39afa5c in term 0.
I20260805 22:48:27.991405 19184 leader_election.cc:302] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73fcd475988c406d9adc1cb03a914246, c32d6164b69140318917f2d1f39afa5c; no voters:
I20260805 22:48:27.991855 19589 raft_consensus.cc:2781] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:48:27.991989 19589 raft_consensus.cc:482] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:48:27.992115 19589 raft_consensus.cc:3037] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:48:27.994423 19589 raft_consensus.cc:504] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } }
I20260805 22:48:27.995042 19589 leader_election.cc:288] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [CANDIDATE]: Term 1 election: Requested vote from peers 73fcd475988c406d9adc1cb03a914246 (127.4.28.1:42475), d8b0e58bfc2d45d29d83658e566d5bb4 (127.4.28.3:42383)
I20260805 22:48:27.995643 19077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1189f8a3f864b908926d123e373120e" candidate_uuid: "c32d6164b69140318917f2d1f39afa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73fcd475988c406d9adc1cb03a914246"
I20260805 22:48:27.995651 19484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1189f8a3f864b908926d123e373120e" candidate_uuid: "c32d6164b69140318917f2d1f39afa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4"
I20260805 22:48:27.995884 19077 raft_consensus.cc:3037] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:48:27.995887 19484 raft_consensus.cc:3037] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:48:27.998366 19077 raft_consensus.cc:2454] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c32d6164b69140318917f2d1f39afa5c in term 1.
I20260805 22:48:27.998379 19484 raft_consensus.cc:2454] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c32d6164b69140318917f2d1f39afa5c in term 1.
I20260805 22:48:27.998832 19186 leader_election.cc:302] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c32d6164b69140318917f2d1f39afa5c, d8b0e58bfc2d45d29d83658e566d5bb4; no voters:
I20260805 22:48:27.999161 19589 raft_consensus.cc:2781] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:48:27.999629 19589 raft_consensus.cc:686] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 1 LEADER]: Becoming Leader. State: Replica: c32d6164b69140318917f2d1f39afa5c, State: Running, Role: LEADER
I20260805 22:48:28.000185 19589 consensus_queue.cc:237] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } }
I20260805 22:48:28.004995 18862 catalog_manager.cc:5495] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c reported cstate change: term changed from 0 to 1, leader changed from <none> to c32d6164b69140318917f2d1f39afa5c (127.4.28.2). New cstate: current_term: 1 leader_uuid: "c32d6164b69140318917f2d1f39afa5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32d6164b69140318917f2d1f39afa5c" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 } health_report { overall_health: UNKNOWN } } }
W20260805 22:48:28.014075 19549 tablet.cc:2369] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:48:28.028156 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.022562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.4.28.2:37713)
0805 22:48:28.022764 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:28.022770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:28.022872 (+ 102us) server_negotiation.cc:407] Connection header received
0805 22:48:28.023074 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:28.023078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:28.023173 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:28.023312 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:48:28.023668 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:28.024458 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:28.025991 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:28.026437 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:28.026552 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:28.027080 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:48:28.027633 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:48:28.027746 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:28.027857 (+ 111us) server_negotiation.cc:299] Negotiation successful
0805 22:48:28.027933 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260805 22:48:28.037458 19077 raft_consensus.cc:1261] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [term 1 FOLLOWER]: Refusing update from remote peer c32d6164b69140318917f2d1f39afa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:48:28.037460 19484 raft_consensus.cc:1261] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [term 1 FOLLOWER]: Refusing update from remote peer c32d6164b69140318917f2d1f39afa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:48:28.038674 19594 consensus_queue.cc:1035] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73fcd475988c406d9adc1cb03a914246" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:48:28.039098 19589 consensus_queue.cc:1035] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b0e58bfc2d45d29d83658e566d5bb4" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:48:28.054013 19599 mvcc.cc:205] Tried to move back new op lower bound from 7315333562509418496 to 7315333562373042176. Current Snapshot: MvccSnapshot[applied={T|T < 7315333562509418496}]
I20260805 22:48:28.067822 18861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:48:28.068177 18861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:48:28.072021 18861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bcd0c3c5fc147309ffefc7bb59ea152: Sending DeleteTablet for 3 replicas of tablet e1189f8a3f864b908926d123e373120e
I20260805 22:48:28.073431 19043 tablet_service.cc:1514] Processing DeleteTablet for tablet e1189f8a3f864b908926d123e373120e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:48:28 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:37570
I20260805 22:48:28.073441 19254 tablet_service.cc:1514] Processing DeleteTablet for tablet e1189f8a3f864b908926d123e373120e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:48:28 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:45842
I20260805 22:48:28.073582 19454 tablet_service.cc:1514] Processing DeleteTablet for tablet e1189f8a3f864b908926d123e373120e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:48:28 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:38254
I20260805 22:48:28.075579 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 18955
I20260805 22:48:28.083691 19614 tablet_replica.cc:330] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: stopping tablet replica
I20260805 22:48:28.085188 19614 raft_consensus.cc:2229] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:48:28.086653 19614 raft_consensus.cc:2258] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:48:28.090749 19614 ts_tablet_manager.cc:1905] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:48:28.096323 19612 tablet_replica.cc:330] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: stopping tablet replica
I20260805 22:48:28.096891 19612 raft_consensus.cc:2229] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:48:28.099645 19612 raft_consensus.cc:2258] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:48:28.101253 19614 ts_tablet_manager.cc:1918] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:48:28.101549 19614 log.cc:1196] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/e1189f8a3f864b908926d123e373120e
Aug 05 22:48:28 dist-test-slave-7lbf krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:48:28.104163 19613 tablet_replica.cc:330] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: stopping tablet replica
I20260805 22:48:28.104241 19614 ts_tablet_manager.cc:1939] T e1189f8a3f864b908926d123e373120e P d8b0e58bfc2d45d29d83658e566d5bb4: Deleting consensus metadata
I20260805 22:48:28.104679 19613 raft_consensus.cc:2229] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:48:28.105439 19612 ts_tablet_manager.cc:1905] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:48:28.105805 19613 raft_consensus.cc:2258] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:48:28.108156 19613 ts_tablet_manager.cc:1905] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:48:28.108392 18845 catalog_manager.cc:4841] TS d8b0e58bfc2d45d29d83658e566d5bb4 (127.4.28.3:42383): tablet e1189f8a3f864b908926d123e373120e (table test-table [id=6ed2624a4a5c4e61ab8f8af8ee956f0e]) successfully deleted
I20260805 22:48:28.114598 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.076452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.4.28.1:42475)
0805 22:48:28.076651 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:28.076658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:28.089632 (+ 12974us) server_negotiation.cc:407] Connection header received
0805 22:48:28.090243 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:28.090251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:28.090356 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:28.090453 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:28.092445 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:28.093765 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:28.097674 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:28.098407 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:28.098525 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:28.105300 (+ 6775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:28.105319 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:28.105337 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:28.105410 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:28.109731 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:28.110847 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:28.110858 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:28.110861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:28.111072 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:28.111580 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:28.111586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:28.111590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:28.112128 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:28.113651 (+ 1523us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:28.113848 (+ 197us) server_negotiation.cc:299] Negotiation successful
0805 22:48:28.114110 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260805 22:48:28.115613 19144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:28.117162 19612 ts_tablet_manager.cc:1918] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:48:28.117545 19612 log.cc:1196] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/e1189f8a3f864b908926d123e373120e
I20260805 22:48:28.118322 19612 ts_tablet_manager.cc:1939] T e1189f8a3f864b908926d123e373120e P c32d6164b69140318917f2d1f39afa5c: Deleting consensus metadata
I20260805 22:48:28.120668 18845 catalog_manager.cc:4841] TS c32d6164b69140318917f2d1f39afa5c (127.4.28.2:37713): tablet e1189f8a3f864b908926d123e373120e (table test-table [id=6ed2624a4a5c4e61ab8f8af8ee956f0e]) successfully deleted
I20260805 22:48:28.237277 19613 ts_tablet_manager.cc:1918] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:48:28.237596 19613 log.cc:1196] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/e1189f8a3f864b908926d123e373120e
I20260805 22:48:28.238166 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 18955
I20260805 22:48:28.238255 19613 ts_tablet_manager.cc:1939] T e1189f8a3f864b908926d123e373120e P 73fcd475988c406d9adc1cb03a914246: Deleting consensus metadata
W20260805 22:48:28.266770 18842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.28.1:42475 (error 108)
I20260805 22:48:28.267364 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 19161
I20260805 22:48:28.269632 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.268979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.0.0.1:37588)
0805 22:48:28.269272 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:28.269403 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260805 22:48:28.269927 18842 catalog_manager.cc:4584] TS 73fcd475988c406d9adc1cb03a914246 (127.4.28.1:42475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1189f8a3f864b908926d123e373120e: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42475: connect: Connection refused (error 111)
Aug 05 22:48:28 dist-test-slave-7lbf krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:48:28.285441 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.268935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.4.28.2:37713)
0805 22:48:28.269099 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:28.269104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:28.269120 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:48:28.269177 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:28.269181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:28.269242 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:28.269330 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:28.270831 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:28.271685 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:28.273041 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:28.273651 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:28.273788 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:28.278311 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:28.278333 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:28.278352 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:28.278412 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:28.281940 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:28.282885 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:28.282894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:28.282897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:28.283057 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:28.283529 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:28.283536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:28.283539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:28.284096 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:28.284483 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:28.284765 (+ 282us) server_negotiation.cc:299] Negotiation successful
0805 22:48:28.285045 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 22:48:28.286387 19339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:28.323782 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.323165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.0.0.1:37596)
0805 22:48:28.323461 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:28.323591 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260805 22:48:28.325073 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.324609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.0.0.1:37612)
0805 22:48:28.324785 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:28.324878 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260805 22:48:28.386032 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.385477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.0.0.1:37624)
0805 22:48:28.385740 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:28.385867 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260805 22:48:28.387252 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.386833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.0.0.1:37636)
0805 22:48:28.386971 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:28.387057 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260805 22:48:28.404069 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 19161
I20260805 22:48:28.430266 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 19355
Aug 05 22:48:28 dist-test-slave-7lbf krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:48:28.447554 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.431246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.4.28.3:42383)
0805 22:48:28.431467 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0805 22:48:28.431474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:48:28.431496 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:48:28.431646 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:48:28.431650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:48:28.431719 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:48:28.431808 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:48:28.433225 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:28.434049 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:28.435418 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:48:28.435833 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:48:28.435955 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:48:28.440833 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:48:28.440855 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:48:28.440876 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:48:28.440933 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:48:28.444018 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:28.445006 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:28.445017 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:28.445027 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:28.445222 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:48:28.445907 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:48:28.445913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:48:28.445915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:48:28.446360 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:48:28.446619 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0805 22:48:28.446980 (+ 361us) server_negotiation.cc:299] Negotiation successful
0805 22:48:28.447196 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260805 22:48:28.448283 19542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:28.468219 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.467687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.0.0.1:37644)
0805 22:48:28.467925 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:28.468050 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260805 22:48:28.469411 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.468953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.0.0.1:37658)
0805 22:48:28.469165 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:28.469266 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260805 22:48:28.537058 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 19355
I20260805 22:48:28.561787 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 18821
I20260805 22:48:28.563043 18906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:48:28.675385 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:48:28.674824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42475 (local address 127.0.0.1:37662)
0805 22:48:28.675077 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0805 22:48:28.675217 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:42475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260805 22:48:28.675383 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 18821
2026-08-05T22:48:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4789 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:48:28 dist-test-slave-7lbf krb5kdc[19628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:48:28 dist-test-slave-7lbf krb5kdc[19628](info): set up 2 sockets
Aug 05 22:48:28 dist-test-slave-7lbf krb5kdc[19628](info): commencing operation
krb5kdc: starting...
W20260805 22:48:29.895170 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:48:30 dist-test-slave-7lbf krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:48:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.28.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.28.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/rangeradmin_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/rangeradmin_127.4.28.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.28.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.28.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/rangerlookup_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/rangerlookup_127.4.28.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.28.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.28.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/HTTP_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/HTTP_127.4.28.20@KRBTEST.COM.keytab.
I20260805 22:48:30.336751 4208 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskW_LE9n/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/postgres -l logfile start
2026-08-05 22:48:32.922 UTC [19677] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-05 22:48:32.922 UTC [19677] LOG: listening on IPv4 address "127.4.28.20", port 52069
2026-08-05 22:48:32.926 UTC [19677] LOG: listening on Unix socket "/tmp/.s.PGSQL.52069"
2026-08-05 22:48:32.950 UTC [19681] LOG: database system was shut down at 2026-08-05 22:48:31 UTC
2026-08-05 22:48:32.954 UTC [19677] LOG: database system is ready to accept connections
I20260805 22:48:34.025022 4208 mini_postgres.cc:96] Postgres bound to 52069
2026-08-05 22:48:34.049 UTC [19698] FATAL: database "slave" does not exist
127.4.28.20:52069 - accepting connections
I20260805 22:48:34.050606 4208 mini_ranger.cc:161] Starting Ranger...
I20260805 22:48:34.085132 4208 mini_ranger.cc:84] Created miniranger Postgres user
I20260805 22:48:34.622803 4208 mini_ranger.cc:87] Created ranger Postgres database
I20260805 22:48:34.622987 4208 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-admin
2026-08-05 22:48:34,937 [I] DB FLAVOR :POSTGRES
2026-08-05 22:48:34,937 [I] --------- Verifying Ranger DB connection ---------
2026-08-05 22:48:34,937 [I] Checking connection..
2026-08-05 22:48:34,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-05 22:48:35,233 [I] Checking connection passed.
2026-08-05 22:48:35,233 [I] --------- Verifying version history table ---------
2026-08-05 22:48:35,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 22:48:35,483 [I] Table x_db_version_h does not exist in database ranger
2026-08-05 22:48:35,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 22:48:35,760 [I] Table x_db_version_h does not exist in database ranger
2026-08-05 22:48:35,760 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-05 22:48:35,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-05 22:48:36.027 UTC [19790] WARNING: there is no transaction in progress
2026-08-05 22:48:36,035 [I] create_dbversion_catalog.sql file imported successfully
2026-08-05 22:48:36,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 22:48:36,304 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-05 22:48:36,305 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-05 22:48:36,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-05 22:48:36,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-05 22:48:36,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7lbf.c.gcp-upstream.internal','N') ;"
2026-08-05 22:48:37,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-05 22:48:37,368 [I] Table x_portal_user does not exist in database ranger
2026-08-05 22:48:37,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-05 22:48:37,612 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-05 22:48:37,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-05 22:48:37,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-05 22:48:38,143 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-05 22:48:38,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-05 22:48:39.206 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.224 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.242 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.259 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.276 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.295 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.362 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.370 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.386 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.397 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.408 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.423 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.434 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.441 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.449 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:39.456 UTC [19947] WARNING: there is no transaction in progress
2026-08-05 22:48:40,209 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-05 22:48:40,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-05 22:48:40,474 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-05 22:48:40,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-05 22:48:40,720 [I] Table x_portal_user already exists in database 'ranger'
2026-08-05 22:48:40,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-05 22:48:40,983 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-05 22:48:40,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-05 22:48:41,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-05 22:48:41,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7lbf.c.gcp-upstream.internal';"
2026-08-05 22:48:41,771 [I] CORE_DB_SCHEMA import status has been updated
2026-08-05 22:48:41,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-05 22:48:42,033 [I] DB_PATCHES have already been applied
I20260805 22:48:42.039480 4208 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-admin:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskW_LE9n/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260805 22:48:42.039614 4208 mini_ranger.cc:193] Using host: 127.4.28.20
I20260805 22:48:42.062768 4208 mini_ranger.cc:227] Ranger admin URL: http://127.4.28.20:40721
Aug 05, 2026 10:48:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 05, 2026 10:48:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 05, 2026 10:48:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-admin/ews/webapp
Aug 05, 2026 10:48:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 05, 2026 10:48:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 05, 2026 10:48:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-admin/ews/webapp].
Aug 05, 2026 10:48:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260805 22:48:43.283757 4208 mini_ranger.cc:160] Time spent starting Ranger: real 9.233s user 0.000s sys 0.076s
Aug 05, 2026 10:49:03 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 05, 2026 10:49:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 05, 2026 10:49:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 05, 2026 10:49:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 05, 2026 10:49:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260805 22:49:05.680586 4208 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.28.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.28.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/rangerkms_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/rangerkms_127.4.28.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.28.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.28.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.28.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/HTTP_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.28.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/HTTP_127.4.28.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 05 22:49:05 dist-test-slave-7lbf krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970145, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260805 22:49:05.914040 4208 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260805 22:49:05.943846 4208 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260805 22:49:06.919145 4208 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260805 22:49:06.919287 4208 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-kms
I20260805 22:49:06.919468 4208 mini_ranger_kms.cc:227] Using postgres at 127.4.28.20:52069
2026-08-05 22:49:07,709 [I] DB FLAVOR :POSTGRES
2026-08-05 22:49:07,709 [I] --------- Verifying Ranger DB connection ---------
2026-08-05 22:49:07,709 [I] Checking connection
2026-08-05 22:49:07,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-05 22:49:07,960 [I] connection success
2026-08-05 22:49:07,961 [I] --------- Verifying Ranger DB tables ---------
2026-08-05 22:49:07,961 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-05 22:49:07,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-05 22:49:08,235 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-05 22:49:08,236 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-05 22:49:08,236 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-05 22:49:08,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:52069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-05 22:49:08,523 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260805 22:49:09.101061 4208 mini_ranger_kms.cc:310] Created kms service
I20260805 22:49:09.333704 4208 mini_ranger_kms.cc:326] Created kudu user
I20260805 22:49:09.385965 4208 mini_ranger_kms.cc:343] Created rangerkms user
Aug 05 22:49:09 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970145, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:49:09.821203 4208 mini_ranger_kms.cc:384] Added ranger policy
I20260805 22:49:09.822728 4208 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-kms:/tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260805 22:49:09.822834 4208 mini_ranger_kms.cc:240] Using host: 127.4.28.20
I20260805 22:49:09.850637 4208 mini_ranger_kms.cc:276] Ranger KMS PID: 20214
I20260805 22:49:09.850875 4208 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.28.20:57847
Aug 05, 2026 10:49:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 05, 2026 10:49:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 05, 2026 10:49:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 05, 2026 10:49:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 05, 2026 10:49:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-kms/ews/webapp].
Aug 05, 2026 10:49:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 05 22:49:12 dist-test-slave-7lbf krb5kdc[19628](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785970152, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.28.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:49:12.353770 4208 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.440s user 0.161s sys 0.440s
I20260805 22:49:12.354203 4208 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260805 22:49:12.354477 4208 mini_ranger_kms.cc:401] 127.4.28.20:57847/kms/v1/keys
Aug 05 22:49:12 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970152, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.28.20@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
Aug 05 22:49:12 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970152, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.28.20@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
Aug 05 22:49:15 dist-test-slave-7lbf krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:49:15.504995 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:33955
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:33955
--ranger_config_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:57847/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:49:16.132831 20282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:49:16.133296 20282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:49:16.133498 20282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:49:16.143064 20282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:49:16.143249 20282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:49:16.143363 20282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:49:16.143481 20282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:49:16.143580 20282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:49:16.157558 20282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.28.20:57847/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:33955
--ranger_config_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:33955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:49:16.159220 20282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:49:16.160925 20282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:49:16.178494 20292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:49:16.178369 20294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:49:16.178431 20296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:49:16.179955 20282 server_base.cc:1034] running on GCE node
I20260805 22:49:16.180981 20282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:49:16.182715 20282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:49:16.184046 20282 hybrid_clock.cc:648] HybridClock initialized: now 1785970156183926 us; error 54 us; skew 500 ppm
I20260805 22:49:16.184638 20282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:49:16 dist-test-slave-7lbf krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970156, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:49:16.189324 20282 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:49:16.191285 20282 webserver.cc:460] Webserver started at http://127.4.28.62:44679/ using document root <none> and password file <none>
I20260805 22:49:16.192050 20282 fs_manager.cc:359] Metadata directory not provided
I20260805 22:49:16.192206 20282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:49:16.192561 20282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 22:49:16 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970156, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:49:16.264142 20282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "0f4a6d9f370b4f99ad0030ba0b57e172"
format_stamp: "Formatted at 2026-08-05 22:49:16 on dist-test-slave-7lbf"
server_key: "5093a1654ccf196cb3fb68ccea2ed394"
server_key_iv: "24be1f33a7ae1e7de38ed1731bad3b0c"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:16.265336 20282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "0f4a6d9f370b4f99ad0030ba0b57e172"
format_stamp: "Formatted at 2026-08-05 22:49:16 on dist-test-slave-7lbf"
server_key: "5093a1654ccf196cb3fb68ccea2ed394"
server_key_iv: "24be1f33a7ae1e7de38ed1731bad3b0c"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:16.273375 20282 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260805 22:49:16.297158 20307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:49:16.298943 20282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:49:16.299188 20282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "0f4a6d9f370b4f99ad0030ba0b57e172"
format_stamp: "Formatted at 2026-08-05 22:49:16 on dist-test-slave-7lbf"
server_key: "5093a1654ccf196cb3fb68ccea2ed394"
server_key_iv: "24be1f33a7ae1e7de38ed1731bad3b0c"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:16.299489 20282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:49:16.316790 20282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:49:16.322757 20282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:49:16.323155 20282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:49:16.359470 20282 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:33955
I20260805 22:49:16.359546 20386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:33955 every 8 connection(s)
I20260805 22:49:16.361393 20282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:49:16.367023 20388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:49:16.371338 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 20282
I20260805 22:49:16.371593 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:49:16.372300 4208 external_mini_cluster.cc:1428] Setting key 7ab98b4f66e5334699d142e6c004f9be
I20260805 22:49:16.385037 20388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172: Bootstrap starting.
I20260805 22:49:16.391222 20388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172: Neither blocks nor log segments found. Creating new log.
Aug 05 22:49:16 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:49:16.393823 20388 log.cc:824] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172: Log is configured to *not* fsync() on all Append() calls
I20260805 22:49:16.399135 20388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172: No bootstrap required, opened a new log
I20260805 22:49:16.405303 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:16.374877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.4.28.62:33955)
0805 22:49:16.378386 (+ 3509us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:16.378400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:16.378464 (+ 64us) server_negotiation.cc:407] Connection header received
0805 22:49:16.382224 (+ 3760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:16.382260 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:16.382803 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:16.383292 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:49:16.384830 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:16.387257 (+ 2427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:16.388737 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:16.389740 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:16.389939 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:16.394247 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:49:16.394258 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:49:16.394278 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:49:16.394332 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:49:16.399395 (+ 5063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:16.400292 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:16.400301 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:16.400309 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:16.400596 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:16.401109 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:16.401113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:16.401116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:16.402063 (+ 947us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:49:16.402565 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:16.403073 (+ 508us) server_negotiation.cc:299] Negotiation successful
0805 22:49:16.403649 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3219,"thread_start_us":2936,"threads_started":1}
I20260805 22:49:16.416431 20388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4a6d9f370b4f99ad0030ba0b57e172" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 33955 } }
I20260805 22:49:16.417237 20388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:49:16.417470 20388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4a6d9f370b4f99ad0030ba0b57e172, State: Initialized, Role: FOLLOWER
I20260805 22:49:16.418493 20388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4a6d9f370b4f99ad0030ba0b57e172" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 33955 } }
I20260805 22:49:16.419018 20388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:49:16.419322 20388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:49:16.419643 20388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:49:16.425028 20388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4a6d9f370b4f99ad0030ba0b57e172" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 33955 } }
I20260805 22:49:16.426039 20388 leader_election.cc:302] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f4a6d9f370b4f99ad0030ba0b57e172; no voters:
I20260805 22:49:16.427418 20388 leader_election.cc:288] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:49:16.427510 20397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:49:16.430233 20397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [term 1 LEADER]: Becoming Leader. State: Replica: 0f4a6d9f370b4f99ad0030ba0b57e172, State: Running, Role: LEADER
I20260805 22:49:16.431679 20388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:49:16.431569 20397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4a6d9f370b4f99ad0030ba0b57e172" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 33955 } }
I20260805 22:49:16.437079 20398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f4a6d9f370b4f99ad0030ba0b57e172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4a6d9f370b4f99ad0030ba0b57e172" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 33955 } } }
I20260805 22:49:16.437228 20399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4a6d9f370b4f99ad0030ba0b57e172. Latest consensus state: current_term: 1 leader_uuid: "0f4a6d9f370b4f99ad0030ba0b57e172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4a6d9f370b4f99ad0030ba0b57e172" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 33955 } } }
I20260805 22:49:16.438832 20398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [sys.catalog]: This master's current role is: LEADER
I20260805 22:49:16.438840 20399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172 [sys.catalog]: This master's current role is: LEADER
I20260805 22:49:16.442183 20410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:49:16.451532 20410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:49:16.452598 20388 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260805 22:49:16.482087 20410 catalog_manager.cc:1347] Generated new cluster ID: 44e35c7efe2b46feb1adb82a46ad088e
I20260805 22:49:16.482278 20410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:49:16.525360 20410 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:49:16.528270 20410 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:49:16.540746 20410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172: Generated new TSK 0
I20260805 22:49:16.542307 20410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:49:17.557653 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:57847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:33955
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:49:17 dist-test-slave-7lbf krb5kdc[19628](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785970157, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260805 22:49:18.145112 20455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:49:18.145556 20455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:49:18.145756 20455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:49:18.156013 20455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:49:18.156273 20455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:49:18.156528 20455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:49:18.177501 20455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.28.20:57847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:49:18.179872 20455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:49:18.181855 20455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:49:18.210472 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:49:18.210702 20470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:49:18.212584 20476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:49:18.214757 20455 server_base.cc:1034] running on GCE node
I20260805 22:49:18.216243 20455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:49:18.219013 20455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:49:18.220485 20455 hybrid_clock.cc:648] HybridClock initialized: now 1785970158220322 us; error 93 us; skew 500 ppm
I20260805 22:49:18.221310 20455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:49:18 dist-test-slave-7lbf krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970158, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:49:18.231055 20455 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:49:18.235539 20455 webserver.cc:460] Webserver started at http://127.4.28.1:34977/ using document root <none> and password file <none>
I20260805 22:49:18.237155 20455 fs_manager.cc:359] Metadata directory not provided
I20260805 22:49:18.237486 20455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:49:18.238229 20455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 22:49:18 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970158, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:49:18.266462 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "3304654381f14c6da19be497ac151329"
format_stamp: "Formatted at 2026-08-05 22:49:18 on dist-test-slave-7lbf"
server_key: "17c23540424595582835cc341c81124c"
server_key_iv: "826e08820ee0cc6695a0ea48adbdbc3a"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:18.268805 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "3304654381f14c6da19be497ac151329"
format_stamp: "Formatted at 2026-08-05 22:49:18 on dist-test-slave-7lbf"
server_key: "17c23540424595582835cc341c81124c"
server_key_iv: "826e08820ee0cc6695a0ea48adbdbc3a"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:18.279899 20455 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260805 22:49:18.303223 20483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:49:18.305574 20455 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260805 22:49:18.305902 20455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "3304654381f14c6da19be497ac151329"
format_stamp: "Formatted at 2026-08-05 22:49:18 on dist-test-slave-7lbf"
server_key: "17c23540424595582835cc341c81124c"
server_key_iv: "826e08820ee0cc6695a0ea48adbdbc3a"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:18.306278 20455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:49:18.342597 20455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:49:18.363077 20455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:49:18.363960 20455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:49:18.372133 20455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:49:18.377485 20455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:49:18.377753 20455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260805 22:49:18.378059 20455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:49:18.378219 20455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:49:18 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970157, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:49:18.693462 20654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:33511 every 8 connection(s)
I20260805 22:49:18.694173 20455 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:33511
I20260805 22:49:18.696646 20455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:49:18.702445 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 20455
I20260805 22:49:18.702706 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:49:18.703575 4208 external_mini_cluster.cc:1428] Setting key 3de81f6a686fbf72021fe61e36ab3866
Aug 05 22:49:18 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970158, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:49:18.770633 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:18.709894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:47797 (local address 127.4.28.62:33955)
0805 22:49:18.740401 (+ 30507us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:18.740409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:18.740443 (+ 34us) server_negotiation.cc:407] Connection header received
0805 22:49:18.743609 (+ 3166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:18.743616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:18.743740 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:18.743844 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:49:18.747749 (+ 3905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:18.749011 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:18.752194 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:18.753042 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:18.753166 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:18.760315 (+ 7149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:49:18.760321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:49:18.760325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:49:18.760426 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:49:18.765073 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:18.766454 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:18.766461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:18.766465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:18.766795 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:18.767740 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:18.767744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:18.767774 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:18.768290 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:49:18.768451 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:18.769640 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0805 22:49:18.769923 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30297,"thread_start_us":30538,"threads_started":1}
I20260805 22:49:18.774106 20658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:18.710160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:33955 (local address 127.4.28.1:47797)
0805 22:49:18.734165 (+ 24005us) negotiation.cc:107] Waiting for socket to connect
0805 22:49:18.734249 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0805 22:49:18.743057 (+ 8808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:49:18.744791 (+ 1734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:49:18.744807 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:49:18.745984 (+ 1177us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:49:18.747408 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:18.747449 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:18.749315 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:18.749332 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:18.751937 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:18.751959 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:18.753860 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:18.753872 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:18.754309 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:18.755404 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:49:18.755460 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:49:18.759957 (+ 4497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:49:18.765286 (+ 5329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:49:18.765299 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:49:18.765334 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:49:18.766234 (+ 900us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:49:18.767037 (+ 803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:49:18.767043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:49:18.767046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:49:18.767569 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:49:18.768453 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:49:18.768463 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:49:18.769040 (+ 577us) client_negotiation.cc:769] Sending connection context
0805 22:49:18.771284 (+ 2244us) client_negotiation.cc:240] Negotiation successful
0805 22:49:18.772077 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23534,"thread_start_us":27239,"threads_started":1}
I20260805 22:49:18.780704 20655 heartbeater.cc:347] Connected to a master server at 127.4.28.62:33955
I20260805 22:49:18.781267 20655 heartbeater.cc:464] Registering TS with master...
I20260805 22:49:18.783051 20655 heartbeater.cc:511] Master 127.4.28.62:33955 requested a full tablet report, sending...
I20260805 22:49:18.787379 20333 ts_manager.cc:194] Registered new tserver with Master: 3304654381f14c6da19be497ac151329 (127.4.28.1:33511)
I20260805 22:49:18.790581 20333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:47797
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
W20260805 22:49:18.868970 4208 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.28.2: real 0.162s user 0.000s sys 0.041s
I20260805 22:49:18.870076 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:57847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:33955
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:49:19 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970157, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
W20260805 22:49:19.775890 20663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:49:19.776362 20663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:49:19.776589 20663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:49:19.788144 20663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:49:19.788417 20663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:49:19.788671 20663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:49:19.795993 20655 heartbeater.cc:503] Master 127.4.28.62:33955 was elected leader, sending a full tablet report...
I20260805 22:49:19.804083 20663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.28.20:57847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:49:19.805846 20663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:49:19.807433 20663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:49:19.829739 20685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:49:19.831959 20686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:49:19.831995 20689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:49:19.833411 20663 server_base.cc:1034] running on GCE node
I20260805 22:49:19.834414 20663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:49:19.835728 20663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:49:19.837030 20663 hybrid_clock.cc:648] HybridClock initialized: now 1785970159836909 us; error 55 us; skew 500 ppm
I20260805 22:49:19.837340 20663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:49:19 dist-test-slave-7lbf krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:49:19.842238 20663 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:49:19.844506 20663 webserver.cc:460] Webserver started at http://127.4.28.2:39273/ using document root <none> and password file <none>
I20260805 22:49:19.845554 20663 fs_manager.cc:359] Metadata directory not provided
I20260805 22:49:19.845793 20663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:49:19.846206 20663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 22:49:19 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:49:19.866015 20663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "975f52bdba18473cb7c2d0f0bd2bdf96"
format_stamp: "Formatted at 2026-08-05 22:49:19 on dist-test-slave-7lbf"
server_key: "defb05ab44539c24eb4a0eb70f209995"
server_key_iv: "48b13a72159c11e286938b6c083b2937"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:19.867614 20663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "975f52bdba18473cb7c2d0f0bd2bdf96"
format_stamp: "Formatted at 2026-08-05 22:49:19 on dist-test-slave-7lbf"
server_key: "defb05ab44539c24eb4a0eb70f209995"
server_key_iv: "48b13a72159c11e286938b6c083b2937"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:19.876613 20663 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s
I20260805 22:49:19.896536 20698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:49:19.898499 20663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260805 22:49:19.898816 20663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "975f52bdba18473cb7c2d0f0bd2bdf96"
format_stamp: "Formatted at 2026-08-05 22:49:19 on dist-test-slave-7lbf"
server_key: "defb05ab44539c24eb4a0eb70f209995"
server_key_iv: "48b13a72159c11e286938b6c083b2937"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:19.899063 20663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:49:19.925104 20663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:49:19.931146 20663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:49:19.931794 20663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:49:19.933691 20663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:49:19.936599 20663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:49:19.936767 20663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:49:19.937009 20663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:49:19.937131 20663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:49:20.046334 20663 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:36177
I20260805 22:49:20.046428 20864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:36177 every 8 connection(s)
I20260805 22:49:20.048377 20663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:49:20.056581 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 20663
I20260805 22:49:20.056792 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:49:20.057205 4208 external_mini_cluster.cc:1428] Setting key f4d12f816e79b60ec160249d250ab3bf
Aug 05 22:49:20 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:49:20.081297 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:20.052939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:33079 (local address 127.4.28.62:33955)
0805 22:49:20.054021 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:20.054026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:20.056143 (+ 2117us) server_negotiation.cc:407] Connection header received
0805 22:49:20.059689 (+ 3546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:20.059695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:20.059803 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:20.059950 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:49:20.063170 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:20.064753 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:20.067486 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:20.068158 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:20.068300 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:20.073140 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:49:20.073147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:49:20.073151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:49:20.073212 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:49:20.076779 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:20.077810 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:20.077815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:20.077818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:20.077998 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:20.078643 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:20.078648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:20.078650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:20.079098 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:49:20.079255 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:20.080360 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0805 22:49:20.080679 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":762,"threads_started":1}
I20260805 22:49:20.082712 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:20.054013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:33955 (local address 127.4.28.2:33079)
0805 22:49:20.055847 (+ 1834us) negotiation.cc:107] Waiting for socket to connect
0805 22:49:20.055901 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 22:49:20.059301 (+ 3400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:49:20.060463 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:49:20.060479 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:49:20.061313 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:49:20.062872 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:20.062908 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:20.065038 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:20.065045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:20.067275 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:20.067294 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:20.068554 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:20.068565 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:20.068849 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:20.069739 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:49:20.069786 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:49:20.072786 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:49:20.076965 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:49:20.076977 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:49:20.076995 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:49:20.077657 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:49:20.078123 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:49:20.078129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:49:20.078133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:49:20.078501 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:49:20.079246 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:49:20.079255 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:49:20.079878 (+ 623us) client_negotiation.cc:769] Sending connection context
0805 22:49:20.080188 (+ 310us) client_negotiation.cc:240] Negotiation successful
0805 22:49:20.080819 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"thread_start_us":1027,"threads_started":1}
I20260805 22:49:20.086464 20867 heartbeater.cc:347] Connected to a master server at 127.4.28.62:33955
I20260805 22:49:20.087018 20867 heartbeater.cc:464] Registering TS with master...
I20260805 22:49:20.088279 20867 heartbeater.cc:511] Master 127.4.28.62:33955 requested a full tablet report, sending...
I20260805 22:49:20.090591 20333 ts_manager.cc:194] Registered new tserver with Master: 975f52bdba18473cb7c2d0f0bd2bdf96 (127.4.28.2:36177)
I20260805 22:49:20.091908 20333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:33079
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:49:20.145382 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:57847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:33955
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:37373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:49:20.926913 20875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:49:20.927405 20875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:49:20.927732 20875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:49:20.939913 20875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:49:20.940150 20875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:49:20.940385 20875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:49:20.954627 20875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.28.20:57847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:33955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:49:20.956493 20875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:49:20.958242 20875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:49:20.978121 20881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:49:20.978466 20885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:49:20.979240 20882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:49:20.983001 20875 server_base.cc:1034] running on GCE node
I20260805 22:49:20.984246 20875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:49:20.986008 20875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:49:20.987366 20875 hybrid_clock.cc:648] HybridClock initialized: now 1785970160987207 us; error 57 us; skew 500 ppm
I20260805 22:49:20.987962 20875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:49:20 dist-test-slave-7lbf krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970160, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:49:20.997231 20875 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:49:21.000170 20875 webserver.cc:460] Webserver started at http://127.4.28.3:33461/ using document root <none> and password file <none>
I20260805 22:49:21.001266 20875 fs_manager.cc:359] Metadata directory not provided
I20260805 22:49:21.001463 20875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:49:21.001947 20875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 22:49:21 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970160, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:49:21.024946 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "11abc0d455b849af9ca3c7b6de241a3a"
format_stamp: "Formatted at 2026-08-05 22:49:21 on dist-test-slave-7lbf"
server_key: "57e7f464d10139e603104c7406c50a62"
server_key_iv: "9dd5ff7600b458092fbbd66d37386fd4"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:21.026407 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "11abc0d455b849af9ca3c7b6de241a3a"
format_stamp: "Formatted at 2026-08-05 22:49:21 on dist-test-slave-7lbf"
server_key: "57e7f464d10139e603104c7406c50a62"
server_key_iv: "9dd5ff7600b458092fbbd66d37386fd4"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:21.035960 20875 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s
I20260805 22:49:21.056092 20896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:49:21.057976 20875 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260805 22:49:21.058220 20875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "11abc0d455b849af9ca3c7b6de241a3a"
format_stamp: "Formatted at 2026-08-05 22:49:21 on dist-test-slave-7lbf"
server_key: "57e7f464d10139e603104c7406c50a62"
server_key_iv: "9dd5ff7600b458092fbbd66d37386fd4"
server_key_version: "kuduclusterkey@0"
I20260805 22:49:21.058528 20875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:49:21.090457 20875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:49:21.095736 20867 heartbeater.cc:503] Master 127.4.28.62:33955 was elected leader, sending a full tablet report...
I20260805 22:49:21.097419 20875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:49:21.098326 20875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:49:21.100402 20875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:49:21.103190 20875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:49:21.103379 20875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:49:21.103643 20875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:49:21.103796 20875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:49:21.210127 20875 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:35485
I20260805 22:49:21.210243 21069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:35485 every 8 connection(s)
I20260805 22:49:21.212141 20875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:49:21.216137 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 20875
I20260805 22:49:21.216341 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:49:21.217033 4208 external_mini_cluster.cc:1428] Setting key 7dcdde4efb2b13cc293a665e2cef2048
Aug 05 22:49:21 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970160, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:49:21.248984 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.217028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:47379 (local address 127.4.28.62:33955)
0805 22:49:21.218342 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.218347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.221002 (+ 2655us) server_negotiation.cc:407] Connection header received
0805 22:49:21.224536 (+ 3534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.224543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.224662 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.225129 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:49:21.227980 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.229181 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.232066 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.233121 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.233285 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.239361 (+ 6076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:49:21.239367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:49:21.239371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:49:21.239427 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:49:21.244059 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.245175 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.245181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.245183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.245409 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.246218 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.246226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.246229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.246831 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:49:21.247118 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.248084 (+ 966us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.248387 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180,"thread_start_us":861,"threads_started":1}
I20260805 22:49:21.250308 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.218236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:33955 (local address 127.4.28.3:47379)
0805 22:49:21.220791 (+ 2555us) negotiation.cc:107] Waiting for socket to connect
0805 22:49:21.220852 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0805 22:49:21.224158 (+ 3306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:49:21.225597 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:49:21.225614 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:49:21.226590 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:49:21.227716 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.227744 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.229457 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.229463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.231783 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.231812 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.233677 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.233690 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.234200 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.235356 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:49:21.235428 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:49:21.238993 (+ 3565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:49:21.244271 (+ 5278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:49:21.244281 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:49:21.244308 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:49:21.244989 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:49:21.245570 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:49:21.245574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:49:21.245577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:49:21.246017 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:49:21.247053 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:49:21.247062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:49:21.247590 (+ 528us) client_negotiation.cc:769] Sending connection context
0805 22:49:21.247985 (+ 395us) client_negotiation.cc:240] Negotiation successful
0805 22:49:21.248690 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2062,"thread_start_us":1642,"threads_started":1}
I20260805 22:49:21.252053 21071 heartbeater.cc:347] Connected to a master server at 127.4.28.62:33955
I20260805 22:49:21.252421 21071 heartbeater.cc:464] Registering TS with master...
I20260805 22:49:21.253448 21071 heartbeater.cc:511] Master 127.4.28.62:33955 requested a full tablet report, sending...
I20260805 22:49:21.255436 20333 ts_manager.cc:194] Registered new tserver with Master: 11abc0d455b849af9ca3c7b6de241a3a (127.4.28.3:35485)
I20260805 22:49:21.256603 20333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:47379
I20260805 22:49:21.261926 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:49:21.293367 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.268998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.4.28.62:33955)
0805 22:49:21.269250 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.269257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.270428 (+ 1171us) server_negotiation.cc:407] Connection header received
0805 22:49:21.270609 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.270615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.270748 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.270882 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:49:21.274062 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.275202 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.278427 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.279250 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.279459 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.285330 (+ 5871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:49:21.285338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:49:21.285342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:49:21.285403 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:49:21.289318 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.290671 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.290676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.290679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.290937 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.291379 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.291383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.291385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.291731 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:49:21.292023 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.292631 (+ 608us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.292936 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260805 22:49:21.298358 20333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:49:21.309055 20333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:49:21.356515 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.327538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.4.28.1:33511)
0805 22:49:21.339404 (+ 11866us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.339415 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.342277 (+ 2862us) server_negotiation.cc:407] Connection header received
0805 22:49:21.342471 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.342481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.342819 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.343079 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:49:21.346443 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.348169 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.353184 (+ 5015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.354866 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.355020 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.355415 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.355719 (+ 304us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.355944 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11603,"thread_start_us":11225,"threads_started":1}
I20260805 22:49:21.356890 21093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.327159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33511 (local address 127.0.0.1:57378)
0805 22:49:21.341885 (+ 14726us) negotiation.cc:107] Waiting for socket to connect
0805 22:49:21.341906 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:49:21.342068 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:49:21.343736 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:49:21.343746 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:49:21.343788 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:49:21.346148 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.346167 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.348320 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.348326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.352953 (+ 4627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.352967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.355158 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.355164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.355374 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.355443 (+ 69us) client_negotiation.cc:769] Sending connection context
0805 22:49:21.355582 (+ 139us) client_negotiation.cc:240] Negotiation successful
0805 22:49:21.355894 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14493,"mutex_wait_us":4,"thread_start_us":12112,"threads_started":1}
I20260805 22:49:21.360226 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.327219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.4.28.3:35485)
0805 22:49:21.341295 (+ 14076us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.341305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.341380 (+ 75us) server_negotiation.cc:407] Connection header received
0805 22:49:21.341560 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.341571 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.341806 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.345026 (+ 3220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:49:21.347690 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.350004 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.354554 (+ 4550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.356628 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.356980 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.357475 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.359294 (+ 1819us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.359507 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13797,"thread_start_us":13498,"threads_started":1}
I20260805 22:49:21.360862 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.326431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:35485 (local address 127.0.0.1:34132)
0805 22:49:21.339910 (+ 13479us) negotiation.cc:107] Waiting for socket to connect
0805 22:49:21.340026 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0805 22:49:21.340309 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:49:21.344958 (+ 4649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:49:21.344963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:49:21.344995 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:49:21.347338 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.347351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.350197 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.350202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.354293 (+ 4091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.354311 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.357272 (+ 2961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.357281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.357617 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.357654 (+ 37us) client_negotiation.cc:769] Sending connection context
0805 22:49:21.357740 (+ 86us) client_negotiation.cc:240] Negotiation successful
0805 22:49:21.357893 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13076,"mutex_wait_us":25,"thread_start_us":14889,"threads_started":1}
I20260805 22:49:21.363221 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.341997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:36177 (local address 127.0.0.1:58254)
0805 22:49:21.347133 (+ 5136us) negotiation.cc:107] Waiting for socket to connect
0805 22:49:21.347154 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:49:21.347288 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:49:21.349481 (+ 2193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:49:21.349485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:49:21.349542 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:49:21.351629 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.351649 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.354578 (+ 2929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.354582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.358793 (+ 4211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.358807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.362347 (+ 3540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.362359 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.362548 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.362587 (+ 39us) client_negotiation.cc:769] Sending connection context
0805 22:49:21.362686 (+ 99us) client_negotiation.cc:240] Negotiation successful
0805 22:49:21.362841 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4931,"thread_start_us":4578,"threads_started":1}
I20260805 22:49:21.364382 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.342214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58254 (local address 127.4.28.2:36177)
0805 22:49:21.348864 (+ 6650us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.348872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.348897 (+ 25us) server_negotiation.cc:407] Connection header received
0805 22:49:21.348980 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.348987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.349228 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.349464 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:49:21.351876 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.354356 (+ 2480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.359023 (+ 4667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.361360 (+ 2337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.361501 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.361969 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.363590 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.363786 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6421,"thread_start_us":4107,"threads_started":1}
I20260805 22:49:21.365983 20970 tablet_service.cc:1467] Processing CreateTablet for tablet 45f5c09793dc445692dfdcee3355182f (DEFAULT_TABLE table=test-table [id=f7633f4e12314538a5cf4ccd81bd94fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:49:21.366912 20552 tablet_service.cc:1467] Processing CreateTablet for tablet 45f5c09793dc445692dfdcee3355182f (DEFAULT_TABLE table=test-table [id=f7633f4e12314538a5cf4ccd81bd94fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:49:21.368603 20970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f5c09793dc445692dfdcee3355182f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:49:21.369114 20552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f5c09793dc445692dfdcee3355182f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:49:21.369447 20770 tablet_service.cc:1467] Processing CreateTablet for tablet 45f5c09793dc445692dfdcee3355182f (DEFAULT_TABLE table=test-table [id=f7633f4e12314538a5cf4ccd81bd94fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:49:21.371493 20770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f5c09793dc445692dfdcee3355182f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:49:21.391835 21102 tablet_bootstrap.cc:492] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: Bootstrap starting.
I20260805 22:49:21.395612 21103 tablet_bootstrap.cc:492] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: Bootstrap starting.
I20260805 22:49:21.399618 21102 tablet_bootstrap.cc:654] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: Neither blocks nor log segments found. Creating new log.
I20260805 22:49:21.402518 21102 log.cc:824] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: Log is configured to *not* fsync() on all Append() calls
I20260805 22:49:21.403030 21103 tablet_bootstrap.cc:654] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: Neither blocks nor log segments found. Creating new log.
I20260805 22:49:21.406091 21103 log.cc:824] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: Log is configured to *not* fsync() on all Append() calls
I20260805 22:49:21.406186 21109 tablet_bootstrap.cc:492] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: Bootstrap starting.
I20260805 22:49:21.408740 21102 tablet_bootstrap.cc:492] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: No bootstrap required, opened a new log
I20260805 22:49:21.409261 21102 ts_tablet_manager.cc:1397] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260805 22:49:21.411043 21103 tablet_bootstrap.cc:492] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: No bootstrap required, opened a new log
I20260805 22:49:21.411460 21103 ts_tablet_manager.cc:1397] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260805 22:49:21.413195 21109 tablet_bootstrap.cc:654] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: Neither blocks nor log segments found. Creating new log.
I20260805 22:49:21.415721 21109 log.cc:824] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: Log is configured to *not* fsync() on all Append() calls
I20260805 22:49:21.423911 21109 tablet_bootstrap.cc:492] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: No bootstrap required, opened a new log
I20260805 22:49:21.424422 21109 ts_tablet_manager.cc:1397] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.010s
I20260805 22:49:21.425283 21103 raft_consensus.cc:348] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:49:21.425876 21103 raft_consensus.cc:374] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:49:21.426034 21103 raft_consensus.cc:729] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975f52bdba18473cb7c2d0f0bd2bdf96, State: Initialized, Role: FOLLOWER
I20260805 22:49:21.426079 21102 raft_consensus.cc:348] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:49:21.426988 21102 raft_consensus.cc:374] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:49:21.426952 21103 consensus_queue.cc:260] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:49:21.427239 21102 raft_consensus.cc:729] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3304654381f14c6da19be497ac151329, State: Initialized, Role: FOLLOWER
I20260805 22:49:21.428119 21102 consensus_queue.cc:260] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:49:21.431109 21102 ts_tablet_manager.cc:1428] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: Time spent starting tablet: real 0.022s user 0.009s sys 0.014s
I20260805 22:49:21.433985 21117 raft_consensus.cc:482] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:49:21.434490 21117 raft_consensus.cc:504] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:49:21.444195 21103 ts_tablet_manager.cc:1428] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: Time spent starting tablet: real 0.033s user 0.019s sys 0.012s
I20260805 22:49:21.444646 21117 leader_election.cc:288] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11abc0d455b849af9ca3c7b6de241a3a (127.4.28.3:35485), 975f52bdba18473cb7c2d0f0bd2bdf96 (127.4.28.2:36177)
I20260805 22:49:21.447543 21109 raft_consensus.cc:348] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:49:21.448362 21109 raft_consensus.cc:374] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:49:21.448608 21109 raft_consensus.cc:729] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11abc0d455b849af9ca3c7b6de241a3a, State: Initialized, Role: FOLLOWER
I20260805 22:49:21.450153 21109 consensus_queue.cc:260] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:49:21.452517 21071 heartbeater.cc:503] Master 127.4.28.62:33955 was elected leader, sending a full tablet report...
I20260805 22:49:21.453311 21109 ts_tablet_manager.cc:1428] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: Time spent starting tablet: real 0.029s user 0.022s sys 0.004s
I20260805 22:49:21.458137 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.440835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:51995 (local address 127.4.28.3:35485)
0805 22:49:21.441096 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.441104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.442743 (+ 1639us) server_negotiation.cc:407] Connection header received
0805 22:49:21.444944 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.444951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.445051 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.445127 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:49:21.446683 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.448274 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.453257 (+ 4983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.455377 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.455526 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.455677 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.457582 (+ 1905us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.457742 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 22:49:21.458384 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.440846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:35485 (local address 127.4.28.1:51995)
0805 22:49:21.442625 (+ 1779us) negotiation.cc:107] Waiting for socket to connect
0805 22:49:21.442647 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:49:21.442923 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:49:21.445402 (+ 2479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:49:21.445407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:49:21.445441 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:49:21.446492 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.446511 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.448483 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.448488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.453044 (+ 4556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.453059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.456904 (+ 3845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.456912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.457319 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.457365 (+ 46us) client_negotiation.cc:769] Sending connection context
0805 22:49:21.457467 (+ 102us) client_negotiation.cc:240] Negotiation successful
0805 22:49:21.457709 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1570,"thread_start_us":1098,"threads_started":1}
I20260805 22:49:21.459992 21004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f5c09793dc445692dfdcee3355182f" candidate_uuid: "3304654381f14c6da19be497ac151329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11abc0d455b849af9ca3c7b6de241a3a" is_pre_election: true
I20260805 22:49:21.461150 21004 raft_consensus.cc:2454] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3304654381f14c6da19be497ac151329 in term 0.
I20260805 22:49:21.462679 20485 leader_election.cc:302] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11abc0d455b849af9ca3c7b6de241a3a, 3304654381f14c6da19be497ac151329; no voters:
I20260805 22:49:21.463217 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.443432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:52535 (local address 127.4.28.2:36177)
0805 22:49:21.443663 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.443668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.451436 (+ 7768us) server_negotiation.cc:407] Connection header received
0805 22:49:21.451805 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.451811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.451889 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.451949 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:49:21.454782 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.456109 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.459607 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.461697 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.461921 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.462090 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.462691 (+ 601us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.462834 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 22:49:21.463449 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.442760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:36177 (local address 127.4.28.1:52535)
0805 22:49:21.451300 (+ 8540us) negotiation.cc:107] Waiting for socket to connect
0805 22:49:21.451327 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0805 22:49:21.451619 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:49:21.453253 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:49:21.453257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:49:21.453290 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:49:21.454460 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.454477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.456288 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.456294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.459428 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:49:21.459443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.462174 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.462183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:49:21.462474 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.462508 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 22:49:21.462610 (+ 102us) client_negotiation.cc:240] Negotiation successful
0805 22:49:21.462754 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8278,"thread_start_us":7934,"threads_started":1}
I20260805 22:49:21.463630 21117 raft_consensus.cc:2781] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:49:21.463951 21117 raft_consensus.cc:482] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:49:21.464107 21117 raft_consensus.cc:3037] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:49:21.465915 21072 tablet.cc:2369] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:49:21.465371 20799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f5c09793dc445692dfdcee3355182f" candidate_uuid: "3304654381f14c6da19be497ac151329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" is_pre_election: true
I20260805 22:49:21.466686 20799 raft_consensus.cc:2454] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3304654381f14c6da19be497ac151329 in term 0.
I20260805 22:49:21.469204 21117 raft_consensus.cc:504] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:49:21.470278 21117 leader_election.cc:288] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [CANDIDATE]: Term 1 election: Requested vote from peers 11abc0d455b849af9ca3c7b6de241a3a (127.4.28.3:35485), 975f52bdba18473cb7c2d0f0bd2bdf96 (127.4.28.2:36177)
I20260805 22:49:21.471031 20799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f5c09793dc445692dfdcee3355182f" candidate_uuid: "3304654381f14c6da19be497ac151329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96"
I20260805 22:49:21.471158 21004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f5c09793dc445692dfdcee3355182f" candidate_uuid: "3304654381f14c6da19be497ac151329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11abc0d455b849af9ca3c7b6de241a3a"
I20260805 22:49:21.471243 20799 raft_consensus.cc:3037] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:49:21.471441 21004 raft_consensus.cc:3037] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:49:21.475270 20799 raft_consensus.cc:2454] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3304654381f14c6da19be497ac151329 in term 1.
I20260805 22:49:21.475323 21004 raft_consensus.cc:2454] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3304654381f14c6da19be497ac151329 in term 1.
I20260805 22:49:21.475961 20485 leader_election.cc:302] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11abc0d455b849af9ca3c7b6de241a3a, 3304654381f14c6da19be497ac151329; no voters:
I20260805 22:49:21.476401 21117 raft_consensus.cc:2781] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:49:21.477406 21117 raft_consensus.cc:686] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 1 LEADER]: Becoming Leader. State: Replica: 3304654381f14c6da19be497ac151329, State: Running, Role: LEADER
I20260805 22:49:21.478184 21117 consensus_queue.cc:237] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:49:21.484634 20333 catalog_manager.cc:5495] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3304654381f14c6da19be497ac151329 (127.4.28.1). New cstate: current_term: 1 leader_uuid: "3304654381f14c6da19be497ac151329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3304654381f14c6da19be497ac151329" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } health_report { overall_health: UNKNOWN } } }
W20260805 22:49:21.494431 20656 tablet.cc:2369] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:49:21.510607 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.502392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.4.28.1:33511)
0805 22:49:21.502657 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.502662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.502681 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:49:21.502787 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.502793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.502991 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.503161 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:49:21.503830 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.504877 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.507154 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.508075 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.508223 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.509104 (+ 881us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:49:21.509773 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:49:21.509959 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.510062 (+ 103us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.510164 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260805 22:49:21.523809 21004 raft_consensus.cc:1261] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [term 1 FOLLOWER]: Refusing update from remote peer 3304654381f14c6da19be497ac151329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:49:21.523813 20799 raft_consensus.cc:1261] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [term 1 FOLLOWER]: Refusing update from remote peer 3304654381f14c6da19be497ac151329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:49:21.525141 21121 consensus_queue.cc:1035] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975f52bdba18473cb7c2d0f0bd2bdf96" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:49:21.525832 21117 consensus_queue.cc:1035] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11abc0d455b849af9ca3c7b6de241a3a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:49:21.543808 21128 mvcc.cc:205] Tried to move back new op lower bound from 7315333781587542016 to 7315333781419569152. Current Snapshot: MvccSnapshot[applied={T|T < 7315333781587542016}]
I20260805 22:49:21.567777 20332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:49:21.568220 20332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:49:21.574955 20332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f4a6d9f370b4f99ad0030ba0b57e172: Sending DeleteTablet for 3 replicas of tablet 45f5c09793dc445692dfdcee3355182f
I20260805 22:49:21.576845 20770 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f5c09793dc445692dfdcee3355182f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:49:21 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:58254
I20260805 22:49:21.576929 20552 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f5c09793dc445692dfdcee3355182f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:49:21 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:57378
I20260805 22:49:21.577955 20970 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f5c09793dc445692dfdcee3355182f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:49:21 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:34132
I20260805 22:49:21.580287 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 20455
I20260805 22:49:21.591042 21142 tablet_replica.cc:330] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: stopping tablet replica
I20260805 22:49:21.591733 21142 raft_consensus.cc:2229] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:49:21.592964 21142 raft_consensus.cc:2258] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:49:21.595542 21143 tablet_replica.cc:330] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: stopping tablet replica
I20260805 22:49:21.595644 21142 ts_tablet_manager.cc:1905] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:49:21.596132 21143 raft_consensus.cc:2229] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:49:21.596829 21147 tablet_replica.cc:330] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: stopping tablet replica
I20260805 22:49:21.597332 21143 raft_consensus.cc:2258] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:49:21.597504 21147 raft_consensus.cc:2229] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:49:21.598358 21147 raft_consensus.cc:2258] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:49:21.600032 21143 ts_tablet_manager.cc:1905] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:49:21.600718 21147 ts_tablet_manager.cc:1905] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:49:21.606504 21142 ts_tablet_manager.cc:1918] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:49:21.606812 21142 log.cc:1196] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/45f5c09793dc445692dfdcee3355182f
I20260805 22:49:21.607755 21142 ts_tablet_manager.cc:1939] T 45f5c09793dc445692dfdcee3355182f P 975f52bdba18473cb7c2d0f0bd2bdf96: Deleting consensus metadata
Aug 05 22:49:21 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:49:21.610572 20309 catalog_manager.cc:4841] TS 975f52bdba18473cb7c2d0f0bd2bdf96 (127.4.28.2:36177): tablet 45f5c09793dc445692dfdcee3355182f (table test-table [id=f7633f4e12314538a5cf4ccd81bd94fe]) successfully deleted
I20260805 22:49:21.611038 21147 ts_tablet_manager.cc:1918] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:49:21.611347 21147 log.cc:1196] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/45f5c09793dc445692dfdcee3355182f
I20260805 22:49:21.612241 21147 ts_tablet_manager.cc:1939] T 45f5c09793dc445692dfdcee3355182f P 11abc0d455b849af9ca3c7b6de241a3a: Deleting consensus metadata
I20260805 22:49:21.615028 20309 catalog_manager.cc:4841] TS 11abc0d455b849af9ca3c7b6de241a3a (127.4.28.3:35485): tablet 45f5c09793dc445692dfdcee3355182f (table test-table [id=f7633f4e12314538a5cf4ccd81bd94fe]) successfully deleted
I20260805 22:49:21.618315 21143 ts_tablet_manager.cc:1918] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:49:21.618580 21143 log.cc:1196] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/45f5c09793dc445692dfdcee3355182f
I20260805 22:49:21.619668 21143 ts_tablet_manager.cc:1939] T 45f5c09793dc445692dfdcee3355182f P 3304654381f14c6da19be497ac151329: Deleting consensus metadata
I20260805 22:49:21.620234 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.582243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.4.28.1:33511)
0805 22:49:21.582469 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.582477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.595641 (+ 13164us) server_negotiation.cc:407] Connection header received
0805 22:49:21.595845 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.595851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.595968 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.596088 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:49:21.598704 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.599830 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.602051 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.602754 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.602885 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.610201 (+ 7316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:49:21.610222 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:49:21.610243 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:49:21.610293 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:49:21.615914 (+ 5621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.616949 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.616959 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.616962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.617246 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.617776 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.617780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.617783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.618340 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:49:21.618861 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.619269 (+ 408us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.619603 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260805 22:49:21.621101 20648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:49:21.730031 20312 catalog_manager.cc:4841] TS 3304654381f14c6da19be497ac151329 (127.4.28.1:33511): tablet 45f5c09793dc445692dfdcee3355182f (table test-table [id=f7633f4e12314538a5cf4ccd81bd94fe]) successfully deleted
I20260805 22:49:21.736850 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 20455
I20260805 22:49:21.759547 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 20663
Aug 05 22:49:21 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:49:21.779618 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.760457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.4.28.2:36177)
0805 22:49:21.760679 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.760684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.760706 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:49:21.760811 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.760816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.760896 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.760995 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:49:21.762419 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.763485 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.764923 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.765760 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.765867 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.771304 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:49:21.771323 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:49:21.771343 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:49:21.771388 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:49:21.776276 (+ 4888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.777043 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.777052 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.777054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.777322 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.777746 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.777750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.777752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.778266 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:49:21.778686 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.778863 (+ 177us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.779094 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260805 22:49:21.780493 20855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:49:21.872409 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 20663
I20260805 22:49:21.895073 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 20875
Aug 05 22:49:21 dist-test-slave-7lbf krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:49:21.911293 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:49:21.895954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.4.28.3:35485)
0805 22:49:21.896151 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0805 22:49:21.896156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:49:21.896176 (+ 20us) server_negotiation.cc:407] Connection header received
0805 22:49:21.896276 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:49:21.896280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:49:21.896349 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:49:21.896434 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:49:21.897875 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.898699 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.900033 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:49:21.900426 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:49:21.900531 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:49:21.905839 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:49:21.905857 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:49:21.905876 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:49:21.905923 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:49:21.908553 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.909256 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.909264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.909267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.909397 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:49:21.909822 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:49:21.909826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:49:21.909828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:49:21.910241 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:49:21.910479 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0805 22:49:21.910751 (+ 272us) server_negotiation.cc:299] Negotiation successful
0805 22:49:21.910935 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 22:49:21.912052 21065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:49:22.004262 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 20875
I20260805 22:49:22.027192 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 20282
I20260805 22:49:22.028584 20380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:49:22.151109 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 20282
I20260805 22:49:22.185794 4208 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260805 22:49:22.522101 4208 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260805 22:49:22.522339 4208 mini_ranger.cc:66] Stopping Ranger...
I20260805 22:49:22.868916 4208 mini_ranger.cc:68] Stopped Ranger
2026-08-05 22:49:22.869 UTC [19677] LOG: received smart shutdown request
2026-08-05 22:49:22.873 UTC [19677] LOG: background worker "logical replication launcher" (PID 19696) exited with exit code 1
2026-08-05 22:49:22.873 UTC [19684] LOG: shutting down
2026-08-05 22:49:23.048 UTC [19677] LOG: database system is shut down
2026-08-05T22:49:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54472 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:49:23 dist-test-slave-7lbf krb5kdc[21165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:49:23 dist-test-slave-7lbf krb5kdc[21165](info): set up 2 sockets
Aug 05 22:49:23 dist-test-slave-7lbf krb5kdc[21165](info): commencing operation
krb5kdc: starting...
W20260805 22:49:24.416021 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:49:24 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:49:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.28.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.28.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/rangeradmin_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/rangeradmin_127.4.28.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.28.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.28.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/rangerlookup_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/rangerlookup_127.4.28.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.28.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.28.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/HTTP_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/HTTP_127.4.28.20@KRBTEST.COM.keytab.
I20260805 22:49:24.828485 4208 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskW_LE9n/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/postgres -l logfile start
2026-08-05 22:49:27.481 UTC [21208] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-05 22:49:27.481 UTC [21208] LOG: listening on IPv4 address "127.4.28.20", port 57275
2026-08-05 22:49:27.485 UTC [21208] LOG: listening on Unix socket "/tmp/.s.PGSQL.57275"
2026-08-05 22:49:27.508 UTC [21211] LOG: database system was shut down at 2026-08-05 22:49:25 UTC
2026-08-05 22:49:27.512 UTC [21208] LOG: database system is ready to accept connections
I20260805 22:49:28.574064 4208 mini_postgres.cc:96] Postgres bound to 57275
2026-08-05 22:49:28.597 UTC [21226] FATAL: database "slave" does not exist
127.4.28.20:57275 - accepting connections
I20260805 22:49:28.598227 4208 mini_ranger.cc:161] Starting Ranger...
I20260805 22:49:28.631222 4208 mini_ranger.cc:84] Created miniranger Postgres user
I20260805 22:49:29.147143 4208 mini_ranger.cc:87] Created ranger Postgres database
I20260805 22:49:29.147248 4208 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-admin
2026-08-05 22:49:29,450 [I] DB FLAVOR :POSTGRES
2026-08-05 22:49:29,451 [I] --------- Verifying Ranger DB connection ---------
2026-08-05 22:49:29,451 [I] Checking connection..
2026-08-05 22:49:29,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-05 22:49:29,729 [I] Checking connection passed.
2026-08-05 22:49:29,729 [I] --------- Verifying version history table ---------
2026-08-05 22:49:29,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 22:49:30,009 [I] Table x_db_version_h does not exist in database ranger
2026-08-05 22:49:30,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 22:49:30,265 [I] Table x_db_version_h does not exist in database ranger
2026-08-05 22:49:30,265 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-05 22:49:30,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-05 22:49:30.539 UTC [21315] WARNING: there is no transaction in progress
2026-08-05 22:49:30,548 [I] create_dbversion_catalog.sql file imported successfully
2026-08-05 22:49:30,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 22:49:30,825 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-05 22:49:30,826 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-05 22:49:30,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-05 22:49:31,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-05 22:49:31,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7lbf.c.gcp-upstream.internal','N') ;"
2026-08-05 22:49:31,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-05 22:49:31,880 [I] Table x_portal_user does not exist in database ranger
2026-08-05 22:49:31,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-05 22:49:32,139 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-05 22:49:32,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-05 22:49:32,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-05 22:49:32,675 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-05 22:49:32,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-05 22:49:33.734 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.753 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.771 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.789 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.807 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.829 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.889 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.897 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.912 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.923 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.934 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.949 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.960 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.968 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.975 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:33.983 UTC [21470] WARNING: there is no transaction in progress
2026-08-05 22:49:34,714 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-05 22:49:34,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-05 22:49:34,966 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-05 22:49:34,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-05 22:49:35,243 [I] Table x_portal_user already exists in database 'ranger'
2026-08-05 22:49:35,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-05 22:49:35,526 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-05 22:49:35,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-05 22:49:35,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-05 22:49:36,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7lbf.c.gcp-upstream.internal';"
2026-08-05 22:49:36,312 [I] CORE_DB_SCHEMA import status has been updated
2026-08-05 22:49:36,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-05 22:49:36,574 [I] DB_PATCHES have already been applied
I20260805 22:49:36.581218 4208 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-admin:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskW_LE9n/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260805 22:49:36.581323 4208 mini_ranger.cc:193] Using host: 127.4.28.20
I20260805 22:49:36.602152 4208 mini_ranger.cc:227] Ranger admin URL: http://127.4.28.20:48521
Aug 05, 2026 10:49:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 05, 2026 10:49:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 05, 2026 10:49:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-admin/ews/webapp
Aug 05, 2026 10:49:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 05, 2026 10:49:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 05, 2026 10:49:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-admin/ews/webapp].
Aug 05, 2026 10:49:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260805 22:49:37.817731 4208 mini_ranger.cc:160] Time spent starting Ranger: real 9.220s user 0.008s sys 0.058s
Aug 05, 2026 10:49:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 05, 2026 10:49:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 05, 2026 10:49:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 05, 2026 10:49:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 05, 2026 10:50:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260805 22:50:01.386973 4208 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.28.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.28.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/rangerkms_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.28.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/rangerkms_127.4.28.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.28.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.28.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.28.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/HTTP_127.4.28.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.28.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/HTTP_127.4.28.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 05 22:50:01 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970201, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260805 22:50:01.607190 4208 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260805 22:50:01.636633 4208 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260805 22:50:02.602413 4208 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260805 22:50:02.602537 4208 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-kms
I20260805 22:50:02.602586 4208 mini_ranger_kms.cc:227] Using postgres at 127.4.28.20:57275
2026-08-05 22:50:02,980 [I] DB FLAVOR :POSTGRES
2026-08-05 22:50:02,980 [I] --------- Verifying Ranger DB connection ---------
2026-08-05 22:50:02,980 [I] Checking connection
2026-08-05 22:50:02,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-05 22:50:03,230 [I] connection success
2026-08-05 22:50:03,231 [I] --------- Verifying Ranger DB tables ---------
2026-08-05 22:50:03,231 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-05 22:50:03,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-05 22:50:03,507 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-05 22:50:03,507 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-05 22:50:03,508 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-05 22:50:03,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.28.20:57275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-05 22:50:03,812 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260805 22:50:04.351066 4208 mini_ranger_kms.cc:310] Created kms service
I20260805 22:50:04.590583 4208 mini_ranger_kms.cc:326] Created kudu user
I20260805 22:50:04.644690 4208 mini_ranger_kms.cc:343] Created rangerkms user
Aug 05 22:50:04 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970201, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:05.110226 4208 mini_ranger_kms.cc:384] Added ranger policy
I20260805 22:50:05.110612 4208 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-kms:/tmp/dist-test-taskW_LE9n/build/asan/bin/postgresql.jar:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskW_LE9n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260805 22:50:05.110678 4208 mini_ranger_kms.cc:240] Using host: 127.4.28.20
I20260805 22:50:05.139400 4208 mini_ranger_kms.cc:276] Ranger KMS PID: 21737
I20260805 22:50:05.139551 4208 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.28.20:60063
Aug 05, 2026 10:50:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 05, 2026 10:50:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 05, 2026 10:50:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 05, 2026 10:50:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 05, 2026 10:50:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-kms/ews/webapp].
Aug 05, 2026 10:50:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260805 22:50:07.651182 4208 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.044s user 0.064s sys 0.353s
I20260805 22:50:07.651409 4208 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260805 22:50:07.651584 4208 mini_ranger_kms.cc:401] 127.4.28.20:60063/kms/v1/keys
Aug 05 22:50:07 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785970207, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.28.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 05 22:50:08 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970207, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.28.20@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
Aug 05 22:50:08 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970207, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.28.20@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
Aug 05 22:50:11 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:50:11.250207 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34279
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:34279
--ranger_config_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:60063/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:11.620684 21805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:11.621011 21805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:11.621102 21805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:11.628587 21805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:11.628813 21805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:11.628916 21805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:11.629000 21805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:11.629066 21805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:11.640720 21805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.28.20:60063/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:34279
--ranger_config_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:11.642215 21805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:11.643651 21805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:11.656401 21811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:11.656473 21812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:11.656433 21815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:11.657115 21805 server_base.cc:1034] running on GCE node
I20260805 22:50:11.658061 21805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:11.659643 21805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:11.660897 21805 hybrid_clock.cc:648] HybridClock initialized: now 1785970211660852 us; error 33 us; skew 500 ppm
I20260805 22:50:11.661294 21805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:11 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970211, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:11.668680 21805 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:50:11.670377 21805 webserver.cc:460] Webserver started at http://127.4.28.62:46217/ using document root <none> and password file <none>
I20260805 22:50:11.671115 21805 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:11.671226 21805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:11.671518 21805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 22:50:11 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970211, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:11.725584 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "73dbd125e77f480c9887f948feb01f89"
format_stamp: "Formatted at 2026-08-05 22:50:11 on dist-test-slave-7lbf"
server_key: "4bc48d293641865cdd083d7b18659061"
server_key_iv: "ec4afb05be9ee6b1cf2d82f6b836ba9e"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:11.726540 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "73dbd125e77f480c9887f948feb01f89"
format_stamp: "Formatted at 2026-08-05 22:50:11 on dist-test-slave-7lbf"
server_key: "4bc48d293641865cdd083d7b18659061"
server_key_iv: "ec4afb05be9ee6b1cf2d82f6b836ba9e"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:11.732641 21805 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260805 22:50:11.750574 21827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:11.752297 21805 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260805 22:50:11.752532 21805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "73dbd125e77f480c9887f948feb01f89"
format_stamp: "Formatted at 2026-08-05 22:50:11 on dist-test-slave-7lbf"
server_key: "4bc48d293641865cdd083d7b18659061"
server_key_iv: "ec4afb05be9ee6b1cf2d82f6b836ba9e"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:11.752764 21805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:11.771435 21805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:11.775980 21805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:11.776335 21805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:11.797724 21805 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:34279
I20260805 22:50:11.797745 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:34279 every 8 connection(s)
I20260805 22:50:11.799436 21805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:50:11.803905 21906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:11.804100 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 21805
I20260805 22:50:11.804344 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:50:11.805225 4208 external_mini_cluster.cc:1428] Setting key 61eea7031c6bac76f7221751324fba4b
I20260805 22:50:11.820489 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Bootstrap starting.
I20260805 22:50:11.825245 21906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:11.826613 21906 log.cc:824] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:11.830654 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: No bootstrap required, opened a new log
Aug 05 22:50:11 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:11.840405 21906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } }
I20260805 22:50:11.840965 21906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:11.841089 21906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73dbd125e77f480c9887f948feb01f89, State: Initialized, Role: FOLLOWER
I20260805 22:50:11.841879 21906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } }
I20260805 22:50:11.842118 21906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:50:11.842254 21906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:50:11.842432 21906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:11.844821 21906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } }
I20260805 22:50:11.845423 21906 leader_election.cc:302] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73dbd125e77f480c9887f948feb01f89; no voters:
I20260805 22:50:11.846154 21906 leader_election.cc:288] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:50:11.846319 21916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:50:11.846575 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:11.807726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.4.28.62:34279)
0805 22:50:11.822468 (+ 14742us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:11.822491 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:11.822556 (+ 65us) server_negotiation.cc:407] Connection header received
0805 22:50:11.825234 (+ 2678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:11.825271 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:11.825921 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:11.826456 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:11.828173 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:11.830231 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:11.831802 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:11.832935 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:11.833179 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:11.837434 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:11.837449 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:11.837475 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:11.837570 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:11.841472 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:11.842333 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:11.842344 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:11.842354 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:11.842611 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:11.843087 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:11.843092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:11.843096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:11.843733 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:11.844075 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:11.844559 (+ 484us) server_negotiation.cc:299] Negotiation successful
0805 22:50:11.845027 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14423,"thread_start_us":15608,"threads_started":1}
I20260805 22:50:11.848809 21916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 1 LEADER]: Becoming Leader. State: Replica: 73dbd125e77f480c9887f948feb01f89, State: Running, Role: LEADER
I20260805 22:50:11.849434 21916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } }
I20260805 22:50:11.850015 21906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:50:11.855545 21917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73dbd125e77f480c9887f948feb01f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } } }
I20260805 22:50:11.856004 21917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:11.855521 21920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73dbd125e77f480c9887f948feb01f89. Latest consensus state: current_term: 1 leader_uuid: "73dbd125e77f480c9887f948feb01f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } } }
I20260805 22:50:11.856374 21920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:11.858165 21935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:50:11.863801 21935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:50:11.864682 21906 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260805 22:50:11.884423 21935 catalog_manager.cc:1347] Generated new cluster ID: 557446c76b674a77bdb9486db702f240
I20260805 22:50:11.884732 21935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:50:11.904095 21935 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:50:11.905655 21935 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:50:11.913461 21935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Generated new TSK 0
I20260805 22:50:11.914438 21935 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:50:12.770156 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:60063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34279
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:50:13 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785970213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260805 22:50:13.260818 21976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:13.261284 21976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:13.261546 21976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:13.272907 21976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:13.273073 21976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:13.273233 21976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:50:13.284276 21976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.28.20:60063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:13.285768 21976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:13.287218 21976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:13.308008 21989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:13.309985 21992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:13.315104 21976 server_base.cc:1034] running on GCE node
W20260805 22:50:13.315640 21995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:13.316632 21976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:13.318921 21976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:13.320256 21976 hybrid_clock.cc:648] HybridClock initialized: now 1785970213320189 us; error 57 us; skew 500 ppm
I20260805 22:50:13.320840 21976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:13 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:13.325966 21976 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:50:13.329619 21976 webserver.cc:460] Webserver started at http://127.4.28.1:33211/ using document root <none> and password file <none>
I20260805 22:50:13.330767 21976 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:13.331030 21976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:13.331496 21976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 22:50:13 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:13.365389 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "83e7dbdf475e47eab97950dda9fed2b2"
format_stamp: "Formatted at 2026-08-05 22:50:13 on dist-test-slave-7lbf"
server_key: "25316dd5517160205b8dc0cc941c8140"
server_key_iv: "ca25da2f572daded59238d9b761552b5"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:13.366791 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "83e7dbdf475e47eab97950dda9fed2b2"
format_stamp: "Formatted at 2026-08-05 22:50:13 on dist-test-slave-7lbf"
server_key: "25316dd5517160205b8dc0cc941c8140"
server_key_iv: "ca25da2f572daded59238d9b761552b5"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:13.375705 21976 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s
I20260805 22:50:13.404258 22009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:13.415421 21976 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.001s
I20260805 22:50:13.415789 21976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "83e7dbdf475e47eab97950dda9fed2b2"
format_stamp: "Formatted at 2026-08-05 22:50:13 on dist-test-slave-7lbf"
server_key: "25316dd5517160205b8dc0cc941c8140"
server_key_iv: "ca25da2f572daded59238d9b761552b5"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:13.416200 21976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:13.459280 21976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:13.469446 21976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:13.470149 21976 kserver.cc:163] Server-wide thread pool size limit: 3276
Aug 05 22:50:13 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:13.484853 21976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:13.506814 21976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:13.507019 21976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260805 22:50:13.507193 21976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:13.507273 21976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:13.589574 21976 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:36465
I20260805 22:50:13.589632 22176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:36465 every 8 connection(s)
I20260805 22:50:13.591570 21976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:50:13.592337 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 21976
I20260805 22:50:13.592552 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:50:13.593380 4208 external_mini_cluster.cc:1428] Setting key 0f1b47ff7b5b4a0a71a7eae6be36ab6a
Aug 05 22:50:13 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
I20260805 22:50:13.632727 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:13.599203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:52049 (local address 127.4.28.62:34279)
0805 22:50:13.612400 (+ 13197us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:13.612407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:13.612435 (+ 28us) server_negotiation.cc:407] Connection header received
0805 22:50:13.612517 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:13.612522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:13.612617 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:13.612745 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:13.615093 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:13.616229 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:13.618919 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:13.619568 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:13.619704 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:13.624440 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:13.624448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:13.624453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:13.624501 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:13.628201 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:13.629410 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:13.629416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:13.629420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:13.629629 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:13.630260 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:13.630265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:13.630269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:13.630700 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:13.630879 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:13.632053 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0805 22:50:13.632323 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11375,"thread_start_us":11179,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
I20260805 22:50:13.635293 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:13.598435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34279 (local address 127.4.28.1:52049)
0805 22:50:13.606326 (+ 7891us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:13.606414 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:13.609191 (+ 2777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:13.613116 (+ 3925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:13.613130 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:13.613879 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:13.614802 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:13.614834 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:13.616482 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:13.616488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:13.618675 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:13.618697 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:13.620077 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:13.620085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:13.620408 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:13.621318 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:13.621366 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:13.624098 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:13.628473 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:13.628484 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:13.628508 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:13.629212 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:13.629792 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:13.629797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:13.629800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:13.630104 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:13.630968 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:13.630979 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:13.631541 (+ 562us) client_negotiation.cc:769] Sending connection context
0805 22:50:13.633075 (+ 1534us) client_negotiation.cc:240] Negotiation successful
0805 22:50:13.633758 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7431,"thread_start_us":10985,"threads_started":1}
I20260805 22:50:13.639468 22178 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34279
I20260805 22:50:13.639844 22178 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:13.640838 22178 heartbeater.cc:511] Master 127.4.28.62:34279 requested a full tablet report, sending...
I20260805 22:50:13.643781 21853 ts_manager.cc:194] Registered new tserver with Master: 83e7dbdf475e47eab97950dda9fed2b2 (127.4.28.1:36465)
I20260805 22:50:13.645965 21853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:52049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:50:13.681406 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:60063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34279
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:50:13 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
W20260805 22:50:14.169394 22187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:14.169783 22187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:14.169905 22187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:14.177433 22187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:14.177616 22187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:14.177754 22187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:50:14.188050 22187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.28.20:60063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:14.189482 22187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:14.190989 22187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:14.205252 22207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:14.205166 22204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:14.205283 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:14.206156 22187 server_base.cc:1034] running on GCE node
I20260805 22:50:14.206828 22187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:14.207935 22187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:14.209175 22187 hybrid_clock.cc:648] HybridClock initialized: now 1785970214209118 us; error 48 us; skew 500 ppm
I20260805 22:50:14.209578 22187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:14 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:14.212956 22187 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:50:14.214610 22187 webserver.cc:460] Webserver started at http://127.4.28.2:37123/ using document root <none> and password file <none>
I20260805 22:50:14.215366 22187 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:14.215488 22187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:14.215814 22187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 22:50:14 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:14.234634 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "6239d85e6e1045cc9fcda6d67b46b6b8"
format_stamp: "Formatted at 2026-08-05 22:50:14 on dist-test-slave-7lbf"
server_key: "3426d4b47a59598f90303e6dd5001afc"
server_key_iv: "13107c9d90326e67bd3dee286c2e5621"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:14.235961 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "6239d85e6e1045cc9fcda6d67b46b6b8"
format_stamp: "Formatted at 2026-08-05 22:50:14 on dist-test-slave-7lbf"
server_key: "3426d4b47a59598f90303e6dd5001afc"
server_key_iv: "13107c9d90326e67bd3dee286c2e5621"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:14.243961 22187 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260805 22:50:14.264055 22222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:14.266187 22187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:50:14.266449 22187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "6239d85e6e1045cc9fcda6d67b46b6b8"
format_stamp: "Formatted at 2026-08-05 22:50:14 on dist-test-slave-7lbf"
server_key: "3426d4b47a59598f90303e6dd5001afc"
server_key_iv: "13107c9d90326e67bd3dee286c2e5621"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:14.266782 22187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:14.296562 22187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:14.302215 22187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:14.302583 22187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:14.303807 22187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:14.305683 22187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:14.305781 22187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:14.305913 22187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:14.305982 22187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:14.339818 22187 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:38439
I20260805 22:50:14.339866 22402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:38439 every 8 connection(s)
I20260805 22:50:14.341226 22187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:50:14.345970 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 22187
I20260805 22:50:14.346159 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:50:14.346782 4208 external_mini_cluster.cc:1428] Setting key 1e0cfe9e507373a5ba1a1447ff2a30d6
Aug 05 22:50:14 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:14.364553 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:14.344253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:43107 (local address 127.4.28.62:34279)
0805 22:50:14.344813 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:14.344818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:14.345899 (+ 1081us) server_negotiation.cc:407] Connection header received
0805 22:50:14.348252 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:14.348259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:14.348379 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:14.348519 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:14.350621 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:14.351456 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:14.353804 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:14.354235 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:14.354347 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:14.358505 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:14.358512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:14.358516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:14.358558 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:14.361144 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:14.362024 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:14.362028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:14.362031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:14.362167 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:14.362650 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:14.362654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:14.362656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:14.362966 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:14.363112 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:14.363909 (+ 797us) server_negotiation.cc:299] Negotiation successful
0805 22:50:14.364165 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":285,"threads_started":1}
I20260805 22:50:14.365669 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:14.344804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34279 (local address 127.4.28.2:43107)
0805 22:50:14.345681 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:14.345750 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:14.347954 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:14.348779 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:14.348789 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:14.349399 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:14.350383 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:14.350413 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:14.351653 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:14.351660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:14.353644 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:14.353663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:14.354510 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:14.354517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:14.354835 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:14.355631 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:14.355670 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:14.358223 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:14.361316 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:14.361327 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:14.361344 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:14.361876 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:14.362286 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:14.362290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:14.362292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:14.362524 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:14.363102 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:14.363111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:14.363491 (+ 380us) client_negotiation.cc:769] Sending connection context
0805 22:50:14.363762 (+ 271us) client_negotiation.cc:240] Negotiation successful
0805 22:50:14.364336 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":331,"threads_started":1}
I20260805 22:50:14.367448 22404 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34279
I20260805 22:50:14.367800 22404 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:14.368681 22404 heartbeater.cc:511] Master 127.4.28.62:34279 requested a full tablet report, sending...
I20260805 22:50:14.370533 21853 ts_manager.cc:194] Registered new tserver with Master: 6239d85e6e1045cc9fcda6d67b46b6b8 (127.4.28.2:38439)
I20260805 22:50:14.371407 21853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:43107
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:50:14.416862 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:60063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34279
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:50:14.650242 22178 heartbeater.cc:503] Master 127.4.28.62:34279 was elected leader, sending a full tablet report...
W20260805 22:50:14.777918 22412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:14.778251 22412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:14.778380 22412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:14.785856 22412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:14.786023 22412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:14.786187 22412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:50:14.798640 22412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.28.20:60063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:14.800151 22412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:14.801616 22412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:14.815378 22422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:14.815445 22423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:14.815416 22426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:14.816280 22412 server_base.cc:1034] running on GCE node
I20260805 22:50:14.816921 22412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:14.818066 22412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:14.819272 22412 hybrid_clock.cc:648] HybridClock initialized: now 1785970214819212 us; error 58 us; skew 500 ppm
I20260805 22:50:14.819594 22412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:14 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:14.822880 22412 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:50:14.824534 22412 webserver.cc:460] Webserver started at http://127.4.28.3:34593/ using document root <none> and password file <none>
I20260805 22:50:14.825280 22412 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:14.825412 22412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:14.825807 22412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 22:50:14 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:14.842638 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "f64ba289292b492aa68b29d6dcb9fa66"
format_stamp: "Formatted at 2026-08-05 22:50:14 on dist-test-slave-7lbf"
server_key: "ee3c92bf18e40467815d75ffd7b7c9de"
server_key_iv: "0028484b01c001e33cdda54da405424f"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:14.843638 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "f64ba289292b492aa68b29d6dcb9fa66"
format_stamp: "Formatted at 2026-08-05 22:50:14 on dist-test-slave-7lbf"
server_key: "ee3c92bf18e40467815d75ffd7b7c9de"
server_key_iv: "0028484b01c001e33cdda54da405424f"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:14.850255 22412 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260805 22:50:14.868811 22439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:14.870860 22412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:50:14.871129 22412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "f64ba289292b492aa68b29d6dcb9fa66"
format_stamp: "Formatted at 2026-08-05 22:50:14 on dist-test-slave-7lbf"
server_key: "ee3c92bf18e40467815d75ffd7b7c9de"
server_key_iv: "0028484b01c001e33cdda54da405424f"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:14.871482 22412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:14.920174 22412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:14.924347 22412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:14.924669 22412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:14.925817 22412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:14.927450 22412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:14.927544 22412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:14.927680 22412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:14.927738 22412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:14.959699 22412 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:33435
I20260805 22:50:14.959746 22615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:33435 every 8 connection(s)
I20260805 22:50:14.961055 22412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:50:14.969780 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 22412
I20260805 22:50:14.969980 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:50:14.970624 4208 external_mini_cluster.cc:1428] Setting key c416b89532ce2e4dab775fd5fd9de3f4
Aug 05 22:50:14 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:14.983111 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:14.963918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:49149 (local address 127.4.28.62:34279)
0805 22:50:14.964491 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:14.964499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:14.965637 (+ 1138us) server_negotiation.cc:407] Connection header received
0805 22:50:14.967535 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:14.967540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:14.967628 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:14.967737 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:14.969250 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:14.970467 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:14.972194 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:14.972797 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:14.972920 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:14.976945 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:14.976953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:14.976959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:14.977006 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:14.979862 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:14.980725 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:14.980730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:14.980732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:14.980925 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:14.981403 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:14.981407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:14.981409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:14.981772 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:14.981923 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:14.982600 (+ 677us) server_negotiation.cc:299] Negotiation successful
0805 22:50:14.982808 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":238,"threads_started":1}
I20260805 22:50:14.984098 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:14.964553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34279 (local address 127.4.28.3:49149)
0805 22:50:14.965397 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:14.965453 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:14.967210 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:14.967959 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:14.967967 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:14.968424 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:14.969071 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:14.969090 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:14.970625 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:14.970629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:14.972027 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:14.972042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:14.973094 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:14.973103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:14.973437 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:14.974318 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:14.974345 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:14.976598 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:14.980043 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:14.980052 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:14.980068 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:14.980586 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:14.981026 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:14.981029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:14.981032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:14.981287 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:14.981877 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:14.981884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:14.982243 (+ 359us) client_negotiation.cc:769] Sending connection context
0805 22:50:14.982497 (+ 254us) client_negotiation.cc:240] Negotiation successful
0805 22:50:14.982922 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":346,"threads_started":1}
I20260805 22:50:14.985687 22616 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34279
I20260805 22:50:14.986033 22616 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:14.986882 22616 heartbeater.cc:511] Master 127.4.28.62:34279 requested a full tablet report, sending...
I20260805 22:50:14.988309 21853 ts_manager.cc:194] Registered new tserver with Master: f64ba289292b492aa68b29d6dcb9fa66 (127.4.28.3:33435)
I20260805 22:50:14.989116 21853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:49149
I20260805 22:50:14.999586 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:50:15.001593 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 21976
Aug 05 22:50:15 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:50:15.020526 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:15.002827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.4.28.1:36465)
0805 22:50:15.003523 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:15.003532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:15.003564 (+ 32us) server_negotiation.cc:407] Connection header received
0805 22:50:15.003633 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:15.003640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:15.003872 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:15.004031 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:15.005724 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:15.006868 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:15.008588 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:15.009197 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:15.009308 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:15.014153 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:15.014173 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:15.014191 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:15.014238 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:15.017258 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:15.018180 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:15.018189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:15.018191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:15.018366 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:15.018803 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:15.018808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:15.018810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:15.019203 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:15.019528 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:15.019850 (+ 322us) server_negotiation.cc:299] Negotiation successful
0805 22:50:15.020151 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":339,"threads_started":1}
I20260805 22:50:15.022018 22169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:15.104876 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 21976
I20260805 22:50:15.131484 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 22187
Aug 05 22:50:15 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:50:15.150038 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:15.132634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.4.28.2:38439)
0805 22:50:15.133264 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:15.133272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:15.133303 (+ 31us) server_negotiation.cc:407] Connection header received
0805 22:50:15.133367 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:15.133374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:15.133660 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:15.133862 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:15.135214 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:15.136314 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:15.137984 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:15.138861 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:15.138996 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:15.143612 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:15.143628 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:15.143647 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:15.143694 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:15.146817 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:15.147679 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:15.147688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:15.147691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:15.147845 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:15.148256 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:15.148260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:15.148262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:15.148659 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:15.148990 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:15.149311 (+ 321us) server_negotiation.cc:299] Negotiation successful
0805 22:50:15.149653 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":350,"threads_started":1}
I20260805 22:50:15.151487 22395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:15.235436 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 22187
I20260805 22:50:15.258729 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 22412
Aug 05 22:50:15 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:50:15.277666 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:15.259866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.4.28.3:33435)
0805 22:50:15.260555 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:15.260563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:15.260597 (+ 34us) server_negotiation.cc:407] Connection header received
0805 22:50:15.260664 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:15.260671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:15.260911 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:15.261053 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:15.262587 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:15.263702 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:15.265283 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:15.266163 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:15.266295 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:15.271179 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:15.271198 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:15.271218 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:15.271265 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:15.274298 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:15.275166 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:15.275175 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:15.275178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:15.275339 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:15.275797 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:15.275801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:15.275803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:15.276207 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:15.276541 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:15.276950 (+ 409us) server_negotiation.cc:299] Negotiation successful
0805 22:50:15.277263 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":343,"threads_started":1}
I20260805 22:50:15.279084 22610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:15.363319 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 22412
I20260805 22:50:15.381901 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 21805
I20260805 22:50:15.383376 21897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:15.492370 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 21805
I20260805 22:50:15.525758 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34279
--webserver_interface=127.4.28.62
--webserver_port=46217
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:34279
--ranger_config_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:60063/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.28.20:60063/kms with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:15.902974 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:15.903364 22630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:15.903520 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:15.911351 22630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:15.911509 22630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:15.911595 22630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:15.911674 22630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:15.911738 22630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:15.922662 22630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.28.20:60063/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:34279
--ranger_config_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=46217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:15.924213 22630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:15.925702 22630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:15.938701 22641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:15.938660 22638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:15.938724 22637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:15.939421 22630 server_base.cc:1034] running on GCE node
I20260805 22:50:15.940326 22630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:15.942026 22630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:15.943279 22630 hybrid_clock.cc:648] HybridClock initialized: now 1785970215943224 us; error 51 us; skew 500 ppm
I20260805 22:50:15.943827 22630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:15 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970215, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:15.947384 22630 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:50:15.949148 22630 webserver.cc:460] Webserver started at http://127.4.28.62:46217/ using document root <none> and password file <none>
I20260805 22:50:15.949990 22630 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:15.950134 22630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 05 22:50:15 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970215, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:16.012590 22630 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260805 22:50:16.016300 22652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:16.018420 22630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260805 22:50:16.018674 22630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "73dbd125e77f480c9887f948feb01f89"
format_stamp: "Formatted at 2026-08-05 22:50:11 on dist-test-slave-7lbf"
server_key: "4bc48d293641865cdd083d7b18659061"
server_key_iv: "ec4afb05be9ee6b1cf2d82f6b836ba9e"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:16.019793 22630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:16.038060 22630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:16.042382 22630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:16.042690 22630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:16.061843 22630 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:34279
I20260805 22:50:16.061879 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:34279 every 8 connection(s)
I20260805 22:50:16.063547 22630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:50:16.067813 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 22630
I20260805 22:50:16.068897 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:60063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:36465
--local_ip_for_outbound_sockets=127.4.28.1
--tserver_master_addrs=127.4.28.62:34279
--webserver_port=33211
--webserver_interface=127.4.28.1
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.28.20:60063/kms with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:50:16.072782 22725 sys_catalog.cc:263] Verifying existing consensus state
I20260805 22:50:16.077598 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Bootstrap starting.
I20260805 22:50:16.098484 22725 log.cc:824] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:16.108459 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:50:16.109030 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Bootstrap complete.
I20260805 22:50:16.118502 22725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } }
I20260805 22:50:16.119431 22725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73dbd125e77f480c9887f948feb01f89, State: Initialized, Role: FOLLOWER
I20260805 22:50:16.120131 22725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } }
I20260805 22:50:16.120369 22725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:50:16.120500 22725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:50:16.120675 22725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:50:16.125379 22725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } }
I20260805 22:50:16.126075 22725 leader_election.cc:302] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73dbd125e77f480c9887f948feb01f89; no voters:
I20260805 22:50:16.126754 22725 leader_election.cc:288] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260805 22:50:16.126930 22731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:50:16.128173 22731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [term 2 LEADER]: Becoming Leader. State: Replica: 73dbd125e77f480c9887f948feb01f89, State: Running, Role: LEADER
I20260805 22:50:16.128805 22731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } }
I20260805 22:50:16.129287 22725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:50:16.131738 22734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73dbd125e77f480c9887f948feb01f89. Latest consensus state: current_term: 2 leader_uuid: "73dbd125e77f480c9887f948feb01f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } } }
I20260805 22:50:16.131793 22732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "73dbd125e77f480c9887f948feb01f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dbd125e77f480c9887f948feb01f89" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34279 } } }
I20260805 22:50:16.132149 22734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:16.132210 22732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:16.143249 22741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:50:16.151368 22741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:50:16.153446 22741 catalog_manager.cc:1259] Loaded cluster ID: 557446c76b674a77bdb9486db702f240
I20260805 22:50:16.153615 22741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:50:16.158509 22741 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:50:16.160912 22741 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Loaded TSK: 0
I20260805 22:50:16.163450 22741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:50:16.168061 22725 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260805 22:50:16.541811 22727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:16.542368 22727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:16.542596 22727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:16.552500 22727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:16.552654 22727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:16.552840 22727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:50:16.565296 22727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.28.20:60063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:36465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=33211
--tserver_master_addrs=127.4.28.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:16.567219 22727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:16.569154 22727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:16.598366 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:16.610054 22777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:16.615327 22727 server_base.cc:1034] running on GCE node
W20260805 22:50:16.615908 22786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:16.616920 22727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:16.618207 22727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:16.619467 22727 hybrid_clock.cc:648] HybridClock initialized: now 1785970216619392 us; error 59 us; skew 500 ppm
I20260805 22:50:16.619838 22727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:16 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970216, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:16.623451 22727 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:50:16.637409 22727 webserver.cc:460] Webserver started at http://127.4.28.1:33211/ using document root <none> and password file <none>
I20260805 22:50:16.638715 22727 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:16.638937 22727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 05 22:50:16 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970216, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:16.668416 22727 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.003s
I20260805 22:50:16.681413 22798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:16.683429 22727 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.008s
I20260805 22:50:16.683662 22727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "83e7dbdf475e47eab97950dda9fed2b2"
format_stamp: "Formatted at 2026-08-05 22:50:13 on dist-test-slave-7lbf"
server_key: "25316dd5517160205b8dc0cc941c8140"
server_key_iv: "ca25da2f572daded59238d9b761552b5"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:16.684772 22727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:16.726778 22727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:16.732646 22727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:16.733006 22727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:16.734609 22727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:16.741590 22727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:16.741748 22727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260805 22:50:16.741881 22727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:16.741968 22727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:16.811335 22727 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:36465
I20260805 22:50:16.811388 22952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:36465 every 8 connection(s)
I20260805 22:50:16.812999 22727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:50:16.822893 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 22727
I20260805 22:50:16.823789 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:60063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:38439
--local_ip_for_outbound_sockets=127.4.28.2
--tserver_master_addrs=127.4.28.62:34279
--webserver_port=37123
--webserver_interface=127.4.28.2
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.28.20:60063/kms with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:50:16 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970216, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:16.847738 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:16.816378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:37233 (local address 127.4.28.62:34279)
0805 22:50:16.817143 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:16.817159 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:16.818308 (+ 1149us) server_negotiation.cc:407] Connection header received
0805 22:50:16.820944 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:16.820959 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:16.821373 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:16.821787 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:16.823732 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:16.826215 (+ 2483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:16.829043 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:16.830449 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:16.830751 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:16.837950 (+ 7199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:16.837961 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:16.837983 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:16.838039 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:16.841365 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:16.842585 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:16.842596 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:16.842603 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:16.842796 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:16.843336 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:16.843342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:16.843345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:16.844195 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:16.844413 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:16.845897 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0805 22:50:16.846436 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":278,"threads_started":1}
I20260805 22:50:16.849997 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:16.816544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34279 (local address 127.4.28.1:37233)
0805 22:50:16.818095 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:16.818147 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:16.820271 (+ 2124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:16.821977 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:16.821989 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:16.822666 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:16.823528 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:16.823560 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:16.826508 (+ 2948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:16.826514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:16.828755 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:16.828782 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:16.830888 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:16.830899 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:16.831195 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:16.835056 (+ 3861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:16.835103 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:16.837577 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:16.841828 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:16.841838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:16.841854 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:16.842384 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:16.842945 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:16.842949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:16.842952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:16.843191 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:16.844532 (+ 1341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:16.844542 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:16.845026 (+ 484us) client_negotiation.cc:769] Sending connection context
0805 22:50:16.845307 (+ 281us) client_negotiation.cc:240] Negotiation successful
0805 22:50:16.848798 (+ 3491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"thread_start_us":1054,"threads_started":1}
I20260805 22:50:16.854048 22953 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34279
I20260805 22:50:16.854429 22953 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:16.855388 22953 heartbeater.cc:511] Master 127.4.28.62:34279 requested a full tablet report, sending...
W20260805 22:50:16.859359 22953 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:34279 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260805 22:50:16.859655 22953 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:16.860025 22953 heartbeater.cc:511] Master 127.4.28.62:34279 requested a full tablet report, sending...
I20260805 22:50:16.861505 22953 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:16.861876 22953 heartbeater.cc:511] Master 127.4.28.62:34279 requested a full tablet report, sending...
W20260805 22:50:16.863147 22953 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 22:50:17.327013 22961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:17.327342 22961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:17.327495 22961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:17.336864 22961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:17.337005 22961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:17.337167 22961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:50:17.348037 22961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.28.20:60063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:38439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=37123
--tserver_master_addrs=127.4.28.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:17.349472 22961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:17.351068 22961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:17.368068 22980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:17.368005 22981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:17.370117 22983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:17.371270 22961 server_base.cc:1034] running on GCE node
I20260805 22:50:17.372012 22961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:17.373186 22961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:17.374537 22961 hybrid_clock.cc:648] HybridClock initialized: now 1785970217374461 us; error 74 us; skew 500 ppm
I20260805 22:50:17.374920 22961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:17 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970217, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:17.378496 22961 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:50:17.380424 22961 webserver.cc:460] Webserver started at http://127.4.28.2:37123/ using document root <none> and password file <none>
I20260805 22:50:17.381258 22961 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:17.381402 22961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 05 22:50:17 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970217, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:17.413364 22961 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.005s sys 0.005s
I20260805 22:50:17.417866 22994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:17.420061 22961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260805 22:50:17.420415 22961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "6239d85e6e1045cc9fcda6d67b46b6b8"
format_stamp: "Formatted at 2026-08-05 22:50:14 on dist-test-slave-7lbf"
server_key: "3426d4b47a59598f90303e6dd5001afc"
server_key_iv: "13107c9d90326e67bd3dee286c2e5621"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:17.422127 22961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:17.456301 22961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:17.462276 22961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:17.462718 22961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:17.474084 22961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:17.476730 22961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:17.476861 22961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:17.477022 22961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:17.477100 22961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:17.530512 22961 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:38439
I20260805 22:50:17.530612 23165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:38439 every 8 connection(s)
I20260805 22:50:17.532681 22961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:50:17.533847 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 22961
I20260805 22:50:17.534677 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.28.20:60063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:33435
--local_ip_for_outbound_sockets=127.4.28.3
--tserver_master_addrs=127.4.28.62:34279
--webserver_port=34593
--webserver_interface=127.4.28.3
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.28.20:60063/kms with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:50:17 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785970217, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 05 22:50:17 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970217, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:17.587200 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:17.550342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34279 (local address 127.4.28.2:37441)
0805 22:50:17.562224 (+ 11882us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:17.562300 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:17.565158 (+ 2858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:17.567467 (+ 2309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:17.567478 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:17.568027 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:17.568695 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:17.568715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:17.570226 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:17.570232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:17.572287 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:17.572310 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:17.573765 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:17.573773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:17.574095 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:17.574892 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:17.574925 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:17.577826 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:17.581926 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:17.581937 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:17.581959 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:17.582574 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:17.583063 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:17.583067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:17.583070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:17.583349 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:17.584143 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:17.584152 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:17.584621 (+ 469us) client_negotiation.cc:769] Sending connection context
0805 22:50:17.584955 (+ 334us) client_negotiation.cc:240] Negotiation successful
0805 22:50:17.585640 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11518,"thread_start_us":11195,"threads_started":1}
I20260805 22:50:17.588760 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:17.551103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:37441 (local address 127.4.28.62:34279)
0805 22:50:17.566043 (+ 14940us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:17.566050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:17.566073 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:50:17.566142 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:17.566148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:17.566241 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:17.566360 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:17.568931 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:17.569990 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:17.572559 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:17.573273 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:17.573408 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:17.578152 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:17.578160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:17.578164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:17.578211 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:17.581695 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:17.582731 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:17.582737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:17.582740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:17.582927 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:17.583499 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:17.583504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:17.583508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:17.583918 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:17.584046 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:17.588088 (+ 4042us) server_negotiation.cc:299] Negotiation successful
0805 22:50:17.588373 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14774,"thread_start_us":15856,"threads_started":1}
I20260805 22:50:17.591301 23166 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34279
I20260805 22:50:17.591646 23166 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:17.592479 23166 heartbeater.cc:511] Master 127.4.28.62:34279 requested a full tablet report, sending...
I20260805 22:50:17.594741 22679 ts_manager.cc:194] Registered new tserver with Master: 6239d85e6e1045cc9fcda6d67b46b6b8 (127.4.28.2:38439)
I20260805 22:50:17.597056 22679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:37441
I20260805 22:50:17.863354 22953 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:17.863951 22953 heartbeater.cc:511] Master 127.4.28.62:34279 requested a full tablet report, sending...
I20260805 22:50:17.865901 22679 ts_manager.cc:194] Registered new tserver with Master: 83e7dbdf475e47eab97950dda9fed2b2 (127.4.28.1:36465)
I20260805 22:50:17.867249 22679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:37233
Aug 05 22:50:17 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970217, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
W20260805 22:50:18.077176 23172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:18.077682 23172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:18.077855 23172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:18.089773 23172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:18.089944 23172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:18.090134 23172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:50:18.102427 23172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.28.20:60063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:33435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=34593
--tserver_master_addrs=127.4.28.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:18.103858 23172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:18.105340 23172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:18.119971 23185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:18.120088 23186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:18.121140 23188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:18.121310 23172 server_base.cc:1034] running on GCE node
I20260805 22:50:18.122094 23172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:18.123235 23172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:18.124476 23172 hybrid_clock.cc:648] HybridClock initialized: now 1785970218124424 us; error 42 us; skew 500 ppm
I20260805 22:50:18.124840 23172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:18 dist-test-slave-7lbf krb5kdc[21165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970218, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:18.129297 23172 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:50:18.135372 23172 webserver.cc:460] Webserver started at http://127.4.28.3:34593/ using document root <none> and password file <none>
I20260805 22:50:18.136585 23172 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:18.136885 23172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 05 22:50:18 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970218, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:18.163684 23172 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260805 22:50:18.167819 23197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:18.169898 23172 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260805 22:50:18.170115 23172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "f64ba289292b492aa68b29d6dcb9fa66"
format_stamp: "Formatted at 2026-08-05 22:50:14 on dist-test-slave-7lbf"
server_key: "ee3c92bf18e40467815d75ffd7b7c9de"
server_key_iv: "0028484b01c001e33cdda54da405424f"
server_key_version: "kuduclusterkey@0"
I20260805 22:50:18.171594 23172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:18.195945 23172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:18.201485 23172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:18.201845 23172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:18.203151 23172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:18.207086 23172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:18.207223 23172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:18.207340 23172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:18.207412 23172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 22:50:18 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970217, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for HTTP/127.4.28.20@KRBTEST.COM
I20260805 22:50:18.333948 23172 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:33435
I20260805 22:50:18.334020 23364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:33435 every 8 connection(s)
I20260805 22:50:18.336221 23172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:50:18.336575 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 23172
Aug 05 22:50:18 dist-test-slave-7lbf krb5kdc[21165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970218, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:18.407554 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.345334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34279 (local address 127.4.28.3:40373)
0805 22:50:18.377550 (+ 32216us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:18.377651 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:18.380741 (+ 3090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:18.383085 (+ 2344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:18.383099 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:18.383877 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:18.384868 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.384902 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.389586 (+ 4684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.389594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.391971 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.391993 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.393455 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.393467 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.393832 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.394680 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:18.394716 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:18.397777 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:18.402007 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:18.402018 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:18.402040 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:18.402706 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:18.403270 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:18.403276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:18.403280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:18.403604 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:18.404430 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:18.404437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:18.404940 (+ 503us) client_negotiation.cc:769] Sending connection context
0805 22:50:18.405286 (+ 346us) client_negotiation.cc:240] Negotiation successful
0805 22:50:18.405963 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31718,"thread_start_us":31349,"threads_started":1}
I20260805 22:50:18.409053 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.381631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:40373 (local address 127.4.28.62:34279)
0805 22:50:18.382337 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.382355 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.382381 (+ 26us) server_negotiation.cc:407] Connection header received
0805 22:50:18.382453 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.382458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.382561 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.387964 (+ 5403us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:18.388188 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.389292 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.392173 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.392922 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.393075 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.398141 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:18.398151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:18.398156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:18.398206 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:18.401782 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.402890 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.402896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.402899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.403111 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.403760 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.403766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.403769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.404186 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:18.404335 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.408402 (+ 4067us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.408657 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":932,"threads_started":1}
I20260805 22:50:18.411027 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.337747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.4.28.62:34279)
0805 22:50:18.378286 (+ 40539us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.378291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.382788 (+ 4497us) server_negotiation.cc:407] Connection header received
0805 22:50:18.382858 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.382863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.382957 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.383063 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:18.385406 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.386606 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.388713 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.389415 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.389560 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.401751 (+ 12191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:18.401760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:18.401765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:18.401802 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:18.406100 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.407163 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.407168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.407171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.407320 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.407863 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.407868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.407871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.408214 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:18.410242 (+ 2028us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.410418 (+ 176us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.410679 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40386,"thread_start_us":43585,"threads_started":1}
I20260805 22:50:18.411478 23365 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34279
I20260805 22:50:18.411906 23365 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:18.412951 23365 heartbeater.cc:511] Master 127.4.28.62:34279 requested a full tablet report, sending...
I20260805 22:50:18.414739 22679 ts_manager.cc:194] Registered new tserver with Master: f64ba289292b492aa68b29d6dcb9fa66 (127.4.28.3:33435)
I20260805 22:50:18.415584 22679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:40373
I20260805 22:50:18.427423 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:50:18.463665 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.441688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.4.28.62:34279)
0805 22:50:18.441949 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.441955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.445816 (+ 3861us) server_negotiation.cc:407] Connection header received
0805 22:50:18.446246 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.446253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.446350 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.446430 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:18.448521 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.449741 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.451530 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.452162 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.452300 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.456065 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:18.456074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:18.456079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:18.456133 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:18.459867 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.460875 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.460882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.460887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.461075 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.461662 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.461668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.461672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.462092 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:18.462325 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.463001 (+ 676us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.463295 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260805 22:50:18.471639 22679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:50:18.483780 22679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:50:18.569725 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.517118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:38439 (local address 127.0.0.1:53804)
0805 22:50:18.546445 (+ 29327us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:18.546531 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:18.546784 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:18.556879 (+ 10095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:18.556885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:18.556916 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:18.557263 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.557276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.560988 (+ 3712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.560994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.565048 (+ 4054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.565070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.568847 (+ 3777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.568853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.569004 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.569029 (+ 25us) client_negotiation.cc:769] Sending connection context
0805 22:50:18.569080 (+ 51us) client_negotiation.cc:240] Negotiation successful
0805 22:50:18.569298 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28913,"thread_start_us":32040,"threads_started":1}
I20260805 22:50:18.569727 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.521059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.4.28.3:33435)
0805 22:50:18.538959 (+ 17900us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.538970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.549027 (+ 10057us) server_negotiation.cc:407] Connection header received
0805 22:50:18.549251 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.549262 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.549597 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.549826 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:18.562407 (+ 12581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.563976 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.566470 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.568022 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.568137 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.568407 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.569211 (+ 804us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.569380 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17670,"thread_start_us":17482,"threads_started":1}
I20260805 22:50:18.570475 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.520662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:33435 (local address 127.0.0.1:54480)
0805 22:50:18.548860 (+ 28198us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:18.548882 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:18.549050 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:18.561619 (+ 12569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:18.561624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:18.561669 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:18.562094 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.562111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.564204 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.564209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.566256 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.566274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.568631 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.568638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.568880 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.568948 (+ 68us) client_negotiation.cc:769] Sending connection context
0805 22:50:18.569967 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0805 22:50:18.570129 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28056,"thread_start_us":26376,"threads_started":1}
I20260805 22:50:18.574749 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.549742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:36465 (local address 127.0.0.1:34668)
0805 22:50:18.555048 (+ 5306us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:18.555069 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:18.555223 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:18.556287 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:18.556295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:18.556342 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:18.556754 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.556774 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.559282 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.559288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.571454 (+ 12166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.571473 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.573988 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.573996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.574194 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.574236 (+ 42us) client_negotiation.cc:769] Sending connection context
0805 22:50:18.574343 (+ 107us) client_negotiation.cc:240] Negotiation successful
0805 22:50:18.574488 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5162,"thread_start_us":5561,"threads_started":1}
I20260805 22:50:18.576478 23265 tablet_service.cc:1467] Processing CreateTablet for tablet dcbe83b55e564c898da9781048c6fdc2 (DEFAULT_TABLE table=test-table [id=c0ae0f3f574b45f7aa5bf4ba23a88748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:18.577557 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.550135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.4.28.1:36465)
0805 22:50:18.554051 (+ 3916us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.554060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.555186 (+ 1126us) server_negotiation.cc:407] Connection header received
0805 22:50:18.555394 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.555405 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.555738 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.555975 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:18.557211 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.559037 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.571700 (+ 12663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.573363 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.573544 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.573801 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.576924 (+ 3123us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.577157 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3728,"thread_start_us":3352,"threads_started":1}
I20260805 22:50:18.580322 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbe83b55e564c898da9781048c6fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:18.585050 22861 tablet_service.cc:1467] Processing CreateTablet for tablet dcbe83b55e564c898da9781048c6fdc2 (DEFAULT_TABLE table=test-table [id=c0ae0f3f574b45f7aa5bf4ba23a88748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:18.587004 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbe83b55e564c898da9781048c6fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:18.588361 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.517172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.4.28.2:38439)
0805 22:50:18.556233 (+ 39061us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.556242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.556271 (+ 29us) server_negotiation.cc:407] Connection header received
0805 22:50:18.556372 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.556380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.556638 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.557460 (+ 822us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:18.557761 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.559311 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.565325 (+ 6014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.568600 (+ 3275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.587609 (+ 19009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.587846 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.587928 (+ 82us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.588101 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38874,"thread_start_us":39724,"threads_started":1}
I20260805 22:50:18.592471 23065 tablet_service.cc:1467] Processing CreateTablet for tablet dcbe83b55e564c898da9781048c6fdc2 (DEFAULT_TABLE table=test-table [id=c0ae0f3f574b45f7aa5bf4ba23a88748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:18.594071 23065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbe83b55e564c898da9781048c6fdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:18.602310 23166 heartbeater.cc:503] Master 127.4.28.62:34279 was elected leader, sending a full tablet report...
I20260805 22:50:18.606086 23407 tablet_bootstrap.cc:492] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: Bootstrap starting.
I20260805 22:50:18.609957 23412 tablet_bootstrap.cc:492] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: Bootstrap starting.
I20260805 22:50:18.610803 23407 tablet_bootstrap.cc:654] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:18.612282 23413 tablet_bootstrap.cc:492] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: Bootstrap starting.
I20260805 22:50:18.612324 23407 log.cc:824] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:18.614351 23412 tablet_bootstrap.cc:654] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:18.615737 23412 log.cc:824] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:18.615998 23413 tablet_bootstrap.cc:654] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:18.616279 23407 tablet_bootstrap.cc:492] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: No bootstrap required, opened a new log
I20260805 22:50:18.616566 23407 ts_tablet_manager.cc:1397] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260805 22:50:18.617431 23413 log.cc:824] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:18.620586 23412 tablet_bootstrap.cc:492] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: No bootstrap required, opened a new log
I20260805 22:50:18.620893 23412 ts_tablet_manager.cc:1397] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260805 22:50:18.623972 23413 tablet_bootstrap.cc:492] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: No bootstrap required, opened a new log
I20260805 22:50:18.624271 23413 ts_tablet_manager.cc:1397] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260805 22:50:18.626843 23407 raft_consensus.cc:348] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } }
I20260805 22:50:18.627339 23407 raft_consensus.cc:374] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:18.627498 23407 raft_consensus.cc:729] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f64ba289292b492aa68b29d6dcb9fa66, State: Initialized, Role: FOLLOWER
I20260805 22:50:18.628327 23407 consensus_queue.cc:260] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } }
I20260805 22:50:18.629750 23365 heartbeater.cc:503] Master 127.4.28.62:34279 was elected leader, sending a full tablet report...
I20260805 22:50:18.630620 23407 ts_tablet_manager.cc:1428] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260805 22:50:18.631739 23412 raft_consensus.cc:348] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } }
I20260805 22:50:18.632180 23412 raft_consensus.cc:374] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:18.632285 23412 raft_consensus.cc:729] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e7dbdf475e47eab97950dda9fed2b2, State: Initialized, Role: FOLLOWER
I20260805 22:50:18.632901 23412 consensus_queue.cc:260] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } }
W20260805 22:50:18.633208 23368 tablet.cc:2369] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:50:18.633172 23413 raft_consensus.cc:348] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } }
I20260805 22:50:18.633641 23413 raft_consensus.cc:374] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:18.633744 23413 raft_consensus.cc:729] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6239d85e6e1045cc9fcda6d67b46b6b8, State: Initialized, Role: FOLLOWER
I20260805 22:50:18.634442 23413 consensus_queue.cc:260] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } }
I20260805 22:50:18.638989 22953 heartbeater.cc:503] Master 127.4.28.62:34279 was elected leader, sending a full tablet report...
I20260805 22:50:18.639319 23413 ts_tablet_manager.cc:1428] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260805 22:50:18.639747 23412 ts_tablet_manager.cc:1428] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: Time spent starting tablet: real 0.019s user 0.012s sys 0.009s
I20260805 22:50:18.641095 23420 raft_consensus.cc:482] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:50:18.641392 23420 raft_consensus.cc:504] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } }
I20260805 22:50:18.643589 23420 leader_election.cc:288] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6239d85e6e1045cc9fcda6d67b46b6b8 (127.4.28.2:38439), 83e7dbdf475e47eab97950dda9fed2b2 (127.4.28.1:36465)
I20260805 22:50:18.650801 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.643823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:38439 (local address 127.4.28.3:51579)
0805 22:50:18.643997 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:18.644019 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:18.644152 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:18.645302 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:18.645306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:18.645335 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:18.645576 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.645588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.646765 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.646771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.648435 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.648451 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.649990 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.649996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.650184 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.650222 (+ 38us) client_negotiation.cc:769] Sending connection context
0805 22:50:18.650326 (+ 104us) client_negotiation.cc:240] Negotiation successful
0805 22:50:18.650484 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260805 22:50:18.650799 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.644809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:51579 (local address 127.4.28.2:38439)
0805 22:50:18.645003 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.645008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.645026 (+ 18us) server_negotiation.cc:407] Connection header received
0805 22:50:18.645081 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.645084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.645152 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.645232 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:18.645763 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.646622 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.648616 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.649819 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.649926 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.650016 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.650381 (+ 365us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.650518 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260805 22:50:18.651767 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.644903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:43551 (local address 127.4.28.1:36465)
0805 22:50:18.645106 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.645110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.645568 (+ 458us) server_negotiation.cc:407] Connection header received
0805 22:50:18.645698 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.645703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.645778 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.645868 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:18.646424 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.647472 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.649590 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.650669 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.650792 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.650894 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.651383 (+ 489us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.651533 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 22:50:18.652364 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.644282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:36465 (local address 127.4.28.3:43551)
0805 22:50:18.645415 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:18.645431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:18.645580 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:18.645873 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:18.645876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:18.645894 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:18.646277 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.646286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.647625 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.647629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.649391 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:18.649407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.651032 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.651038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:18.651196 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.651226 (+ 30us) client_negotiation.cc:769] Sending connection context
0805 22:50:18.651953 (+ 727us) client_negotiation.cc:240] Negotiation successful
0805 22:50:18.652085 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":1,"thread_start_us":565,"threads_started":1}
I20260805 22:50:18.652251 23101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbe83b55e564c898da9781048c6fdc2" candidate_uuid: "f64ba289292b492aa68b29d6dcb9fa66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" is_pre_election: true
I20260805 22:50:18.652910 23101 raft_consensus.cc:2454] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f64ba289292b492aa68b29d6dcb9fa66 in term 0.
I20260805 22:50:18.653765 23204 leader_election.cc:302] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6239d85e6e1045cc9fcda6d67b46b6b8, f64ba289292b492aa68b29d6dcb9fa66; no voters:
I20260805 22:50:18.654358 23420 raft_consensus.cc:2781] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:50:18.654534 23420 raft_consensus.cc:482] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:50:18.654640 23420 raft_consensus.cc:3037] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:18.655315 22886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbe83b55e564c898da9781048c6fdc2" candidate_uuid: "f64ba289292b492aa68b29d6dcb9fa66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e7dbdf475e47eab97950dda9fed2b2" is_pre_election: true
I20260805 22:50:18.655925 22886 raft_consensus.cc:2454] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f64ba289292b492aa68b29d6dcb9fa66 in term 0.
I20260805 22:50:18.657326 23420 raft_consensus.cc:504] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } }
I20260805 22:50:18.657966 23420 leader_election.cc:288] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [CANDIDATE]: Term 1 election: Requested vote from peers 6239d85e6e1045cc9fcda6d67b46b6b8 (127.4.28.2:38439), 83e7dbdf475e47eab97950dda9fed2b2 (127.4.28.1:36465)
I20260805 22:50:18.658519 22886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbe83b55e564c898da9781048c6fdc2" candidate_uuid: "f64ba289292b492aa68b29d6dcb9fa66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e7dbdf475e47eab97950dda9fed2b2"
I20260805 22:50:18.658529 23101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbe83b55e564c898da9781048c6fdc2" candidate_uuid: "f64ba289292b492aa68b29d6dcb9fa66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8"
I20260805 22:50:18.658699 22886 raft_consensus.cc:3037] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:18.658725 23101 raft_consensus.cc:3037] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:18.660902 22886 raft_consensus.cc:2454] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f64ba289292b492aa68b29d6dcb9fa66 in term 1.
I20260805 22:50:18.660955 23101 raft_consensus.cc:2454] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f64ba289292b492aa68b29d6dcb9fa66 in term 1.
I20260805 22:50:18.661334 23204 leader_election.cc:302] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e7dbdf475e47eab97950dda9fed2b2, f64ba289292b492aa68b29d6dcb9fa66; no voters:
I20260805 22:50:18.661672 23420 raft_consensus.cc:2781] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:50:18.662146 23420 raft_consensus.cc:686] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 1 LEADER]: Becoming Leader. State: Replica: f64ba289292b492aa68b29d6dcb9fa66, State: Running, Role: LEADER
I20260805 22:50:18.662634 23420 consensus_queue.cc:237] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } }
I20260805 22:50:18.667318 22679 catalog_manager.cc:5495] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 reported cstate change: term changed from 0 to 1, leader changed from <none> to f64ba289292b492aa68b29d6dcb9fa66 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "f64ba289292b492aa68b29d6dcb9fa66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f64ba289292b492aa68b29d6dcb9fa66" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 33435 } health_report { overall_health: HEALTHY } } }
I20260805 22:50:18.684615 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.678942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.4.28.3:33435)
0805 22:50:18.679129 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.679135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.679186 (+ 51us) server_negotiation.cc:407] Connection header received
0805 22:50:18.679455 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.679460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.679554 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.679691 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:50:18.680104 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.680848 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.682350 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.682776 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.682903 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.683456 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:50:18.683997 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:50:18.684148 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.684259 (+ 111us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.684350 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 22:50:18.694082 22886 raft_consensus.cc:1261] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [term 1 FOLLOWER]: Refusing update from remote peer f64ba289292b492aa68b29d6dcb9fa66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:18.694082 23101 raft_consensus.cc:1261] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [term 1 FOLLOWER]: Refusing update from remote peer f64ba289292b492aa68b29d6dcb9fa66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:18.695344 23420 consensus_queue.cc:1035] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e7dbdf475e47eab97950dda9fed2b2" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:18.695788 23424 consensus_queue.cc:1035] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6239d85e6e1045cc9fcda6d67b46b6b8" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 38439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:18.725033 22677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:50:18.725623 22677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:50:18.730352 22677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73dbd125e77f480c9887f948feb01f89: Sending DeleteTablet for 3 replicas of tablet dcbe83b55e564c898da9781048c6fdc2
I20260805 22:50:18.731949 22861 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbe83b55e564c898da9781048c6fdc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:18 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:34668
I20260805 22:50:18.731985 23265 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbe83b55e564c898da9781048c6fdc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:18 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:54480
I20260805 22:50:18.732990 23443 tablet_replica.cc:330] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: stopping tablet replica
I20260805 22:50:18.733034 23065 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbe83b55e564c898da9781048c6fdc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:18 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:53804
I20260805 22:50:18.733489 23443 raft_consensus.cc:2229] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:50:18.734253 23443 raft_consensus.cc:2258] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:18.735100 23446 tablet_replica.cc:330] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: stopping tablet replica
I20260805 22:50:18.735563 23446 raft_consensus.cc:2229] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:50:18.736205 23446 raft_consensus.cc:2258] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:18.738754 23446 ts_tablet_manager.cc:1905] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:18.738979 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 22727
I20260805 22:50:18.741091 23443 ts_tablet_manager.cc:1905] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:18.745855 23446 ts_tablet_manager.cc:1918] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:18.746085 23446 log.cc:1196] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/dcbe83b55e564c898da9781048c6fdc2
I20260805 22:50:18.746651 23446 ts_tablet_manager.cc:1939] T dcbe83b55e564c898da9781048c6fdc2 P 83e7dbdf475e47eab97950dda9fed2b2: Deleting consensus metadata
I20260805 22:50:18.747116 23447 tablet_replica.cc:330] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: stopping tablet replica
I20260805 22:50:18.747763 23447 raft_consensus.cc:2229] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:50:18.748803 22655 catalog_manager.cc:4841] TS 83e7dbdf475e47eab97950dda9fed2b2 (127.4.28.1:36465): tablet dcbe83b55e564c898da9781048c6fdc2 (table test-table [id=c0ae0f3f574b45f7aa5bf4ba23a88748]) successfully deleted
I20260805 22:50:18.748348 23447 raft_consensus.cc:2258] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:18.751427 23447 ts_tablet_manager.cc:1905] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:18.755192 23443 ts_tablet_manager.cc:1918] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:18.756582 23443 log.cc:1196] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/dcbe83b55e564c898da9781048c6fdc2
I20260805 22:50:18.757815 23447 ts_tablet_manager.cc:1918] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:18.757999 23447 log.cc:1196] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/dcbe83b55e564c898da9781048c6fdc2
I20260805 22:50:18.758492 23447 ts_tablet_manager.cc:1939] T dcbe83b55e564c898da9781048c6fdc2 P 6239d85e6e1045cc9fcda6d67b46b6b8: Deleting consensus metadata
I20260805 22:50:18.760144 22655 catalog_manager.cc:4841] TS 6239d85e6e1045cc9fcda6d67b46b6b8 (127.4.28.2:38439): tablet dcbe83b55e564c898da9781048c6fdc2 (table test-table [id=c0ae0f3f574b45f7aa5bf4ba23a88748]) successfully deleted
I20260805 22:50:18.760735 23443 ts_tablet_manager.cc:1939] T dcbe83b55e564c898da9781048c6fdc2 P f64ba289292b492aa68b29d6dcb9fa66: Deleting consensus metadata
I20260805 22:50:18.762856 22659 catalog_manager.cc:4841] TS f64ba289292b492aa68b29d6dcb9fa66 (127.4.28.3:33435): tablet dcbe83b55e564c898da9781048c6fdc2 (table test-table [id=c0ae0f3f574b45f7aa5bf4ba23a88748]) successfully deleted
I20260805 22:50:18.763481 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.742018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.4.28.1:36465)
0805 22:50:18.742217 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.742223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.742246 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:50:18.742315 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.742320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.742407 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.742517 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:18.744761 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.746195 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.749412 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.750120 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.750262 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.755565 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:18.755588 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:18.755608 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:18.755663 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:18.759903 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.760967 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.760976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.760978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.761126 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.761601 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.761607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.761611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.762048 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:18.762357 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.762894 (+ 537us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.763151 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:50:18.764287 22946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:18.859010 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 22727
I20260805 22:50:18.891371 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 22961
I20260805 22:50:18.906028 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:18.892281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.4.28.2:38439)
0805 22:50:18.892511 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:18.892517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:18.892538 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:50:18.892609 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:18.892615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:18.892709 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:18.892800 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:18.894310 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.895084 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.896486 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:18.896929 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:18.897050 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:18.899824 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:18.899841 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:18.899862 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:18.899906 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:18.902945 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.903755 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.903764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.903767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.903918 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:18.904333 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:18.904338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:18.904340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:18.904742 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:18.905055 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:18.905397 (+ 342us) server_negotiation.cc:299] Negotiation successful
0805 22:50:18.905653 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260805 22:50:18.906884 23160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:18.997555 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 22961
I20260805 22:50:19.024646 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 23172
I20260805 22:50:19.042832 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:19.025657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.4.28.3:33435)
0805 22:50:19.025864 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:19.025870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:19.025894 (+ 24us) server_negotiation.cc:407] Connection header received
0805 22:50:19.026002 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:19.026008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:19.026081 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:19.026168 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:19.027592 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:19.028899 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:19.030551 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:19.031291 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:19.031476 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:19.034466 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:19.034493 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:19.034522 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:19.034583 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:19.039011 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:19.039967 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:19.039979 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:19.039983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:19.040190 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:19.040657 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:19.040662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:19.040666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:19.041187 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:19.041587 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:19.042152 (+ 565us) server_negotiation.cc:299] Negotiation successful
0805 22:50:19.042453 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:50:19.043736 23357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:19.167017 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 23172
I20260805 22:50:19.191654 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 22630
I20260805 22:50:19.193012 22718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:19.339008 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 22630
I20260805 22:50:19.382157 4208 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260805 22:50:19.733917 4208 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260805 22:50:19.734165 4208 mini_ranger.cc:66] Stopping Ranger...
I20260805 22:50:20.081238 4208 mini_ranger.cc:68] Stopped Ranger
2026-08-05 22:50:20.081 UTC [21208] LOG: received smart shutdown request
2026-08-05 22:50:20.084 UTC [21208] LOG: background worker "logical replication launcher" (PID 21222) exited with exit code 1
2026-08-05 22:50:20.085 UTC [21213] LOG: shutting down
2026-08-05 22:50:20.260 UTC [21208] LOG: database system is shut down
2026-08-05T22:50:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57218 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:50:20 dist-test-slave-7lbf krb5kdc[23463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:50:20 dist-test-slave-7lbf krb5kdc[23463](info): set up 2 sockets
Aug 05 22:50:20 dist-test-slave-7lbf krb5kdc[23463](info): commencing operation
krb5kdc: starting...
W20260805 22:50:21.593658 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.004s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:50:21 dist-test-slave-7lbf krb5kdc[23463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:50:21.913211 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37627
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:37627,127.4.28.61:44033,127.4.28.60:45529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:22.288771 23488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:22.289134 23488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:22.289259 23488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:22.296926 23488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:22.297068 23488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:22.297123 23488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:22.297206 23488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:22.297267 23488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:22.307823 23488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:37627,127.4.28.61:44033,127.4.28.60:45529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:22.309356 23488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:22.310875 23488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:22.324087 23499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:22.324115 23502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:22.324142 23498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:22.327208 23488 server_base.cc:1034] running on GCE node
I20260805 22:50:22.328158 23488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:22.329807 23488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:22.331079 23488 hybrid_clock.cc:648] HybridClock initialized: now 1785970222331021 us; error 46 us; skew 500 ppm
I20260805 22:50:22.331480 23488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:22 dist-test-slave-7lbf krb5kdc[23463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970222, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:22.339460 23488 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:50:22.341285 23488 webserver.cc:460] Webserver started at http://127.4.28.62:38477/ using document root <none> and password file <none>
I20260805 22:50:22.342135 23488 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:22.342259 23488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:22.342599 23488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:22.345988 23488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6"
format_stamp: "Formatted at 2026-08-05 22:50:22 on dist-test-slave-7lbf"
server_key: "9aab1bf60d26c107d2ad9349645e4058"
server_key_iv: "f29af83777e9f844c917e4b44e5a693e"
server_key_version: "encryptionkey@0"
I20260805 22:50:22.346864 23488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6"
format_stamp: "Formatted at 2026-08-05 22:50:22 on dist-test-slave-7lbf"
server_key: "9aab1bf60d26c107d2ad9349645e4058"
server_key_iv: "f29af83777e9f844c917e4b44e5a693e"
server_key_version: "encryptionkey@0"
I20260805 22:50:22.352722 23488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:50:22.356809 23510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:22.358528 23488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:50:22.358753 23488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6"
format_stamp: "Formatted at 2026-08-05 22:50:22 on dist-test-slave-7lbf"
server_key: "9aab1bf60d26c107d2ad9349645e4058"
server_key_iv: "f29af83777e9f844c917e4b44e5a693e"
server_key_version: "encryptionkey@0"
I20260805 22:50:22.359005 23488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:22.377381 23488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:22.382416 23488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:22.382747 23488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:22.402244 23488 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:37627
I20260805 22:50:22.402276 23577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:37627 every 8 connection(s)
I20260805 22:50:22.403789 23488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:50:22.407266 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 23488
I20260805 22:50:22.407531 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:50:22.408423 4208 external_mini_cluster.cc:1428] Setting key b08131dc270ceb2df887b9634e746a72
I20260805 22:50:22.408702 23579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:22.419090 23579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:50:22 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970222, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.28.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.61@KRBTEST.COM" created.
I20260805 22:50:22.449609 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.423561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37627 (local address 127.0.0.1:46584)
0805 22:50:22.425453 (+ 1892us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.425504 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:22.428098 (+ 2594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:22.429277 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:22.429285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:22.430220 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:22.430871 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:22.430907 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.432747 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.432752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:22.434667 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:22.434690 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.435905 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.435913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:22.436093 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:22.436927 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:22.436965 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:22.440269 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:22.444300 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:22.444317 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:22.444330 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:22.445014 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:22.445710 (+ 696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:22.445715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:22.445717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:22.445964 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:22.447131 (+ 1167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:22.447139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:22.447411 (+ 272us) client_negotiation.cc:769] Sending connection context
0805 22:50:22.447703 (+ 292us) client_negotiation.cc:240] Negotiation successful
0805 22:50:22.448209 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1538,"thread_start_us":562,"threads_started":1}
I20260805 22:50:22.449645 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.423561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.4.28.62:37627)
0805 22:50:22.425338 (+ 1777us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:22.425351 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:22.425679 (+ 328us) server_negotiation.cc:407] Connection header received
0805 22:50:22.428650 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:22.428658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:22.428967 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:22.429173 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:22.431152 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.432536 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.434876 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.435585 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.435759 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:22.440653 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:22.440665 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:22.440683 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:22.440728 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:22.444078 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:22.445222 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:22.445234 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:22.445237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:22.445558 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:22.446109 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:22.446114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:22.446117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:22.446929 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:22.447160 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:22.447891 (+ 731us) server_negotiation.cc:299] Negotiation successful
0805 22:50:22.448406 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":188,"server-negotiator.queue_time_us":1565,"spinlock_wait_cycles":62848,"thread_start_us":590,"threads_started":1}
I20260805 22:50:22.453436 23579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:50:22.454887 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.454404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34862)
0805 22:50:22.454612 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.454740 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260805 22:50:22.455242 23513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
I20260805 22:50:22.457161 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.456678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34864)
0805 22:50:22.456889 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.456976 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260805 22:50:22.457398 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:44033: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab.
Entry for principal kudu/127.4.28.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab.
I20260805 22:50:22.481469 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.61:44033
--webserver_interface=127.4.28.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab
--principal=kudu/127.4.28.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:37627,127.4.28.61:44033,127.4.28.60:45529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:50:22.506927 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } attempt: 1
I20260805 22:50:22.508404 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.507789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34878)
0805 22:50:22.508050 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.508182 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260805 22:50:22.509840 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.509257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34884)
0805 22:50:22.509460 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.509620 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260805 22:50:22.510131 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:44033: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
I20260805 22:50:22.578454 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } attempt: 2
I20260805 22:50:22.579891 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.579313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34900)
0805 22:50:22.579630 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.579732 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260805 22:50:22.581036 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.580637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34908)
0805 22:50:22.580795 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.580891 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260805 22:50:22.581246 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:44033: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
I20260805 22:50:22.672537 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } attempt: 3
I20260805 22:50:22.673796 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.673262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34922)
0805 22:50:22.673495 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.673646 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260805 22:50:22.674940 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.674534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34928)
0805 22:50:22.674695 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.674780 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260805 22:50:22.675153 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:44033: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
I20260805 22:50:22.818459 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } attempt: 4
I20260805 22:50:22.819844 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.819218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34940)
0805 22:50:22.819579 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.819681 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260805 22:50:22.821331 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.820870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34948)
0805 22:50:22.821053 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.821173 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260805 22:50:22.821584 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:44033: Network error: Client connection negotiation failed: client connection to 127.4.28.61:44033: connect: Connection refused (error 111)
W20260805 22:50:22.847608 23596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:22.847944 23596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:22.848059 23596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:22.855726 23596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:22.855885 23596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:22.855985 23596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:22.856073 23596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:22.856138 23596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:22.866948 23596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:37627,127.4.28.61:44033,127.4.28.60:45529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab
--principal=kudu/127.4.28.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.61:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.28.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:22.868412 23596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:22.869902 23596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:22.882598 23604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:22.882639 23608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:22.882563 23605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:22.883267 23596 server_base.cc:1034] running on GCE node
I20260805 22:50:22.884265 23596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:22.885958 23596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:22.887194 23596 hybrid_clock.cc:648] HybridClock initialized: now 1785970222887140 us; error 50 us; skew 500 ppm
I20260805 22:50:22.887581 23596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:22 dist-test-slave-7lbf krb5kdc[23463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970222, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:22.891202 23596 init.cc:377] Logged in from keytab as kudu/127.4.28.61@KRBTEST.COM (short username kudu)
I20260805 22:50:22.892974 23596 webserver.cc:460] Webserver started at http://127.4.28.61:45531/ using document root <none> and password file <none>
I20260805 22:50:22.893843 23596 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:22.893983 23596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:22.894316 23596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:22.897727 23596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/data/instance:
uuid: "aef6b5782a784ac2b64e9c5c94b4033c"
format_stamp: "Formatted at 2026-08-05 22:50:22 on dist-test-slave-7lbf"
server_key: "b9c5359775cda36712565b08849a7d32"
server_key_iv: "18a69db6084c0cec2159a7ea4ee6b68b"
server_key_version: "encryptionkey@0"
I20260805 22:50:22.898635 23596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/wal/instance:
uuid: "aef6b5782a784ac2b64e9c5c94b4033c"
format_stamp: "Formatted at 2026-08-05 22:50:22 on dist-test-slave-7lbf"
server_key: "b9c5359775cda36712565b08849a7d32"
server_key_iv: "18a69db6084c0cec2159a7ea4ee6b68b"
server_key_version: "encryptionkey@0"
I20260805 22:50:22.904340 23596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260805 22:50:22.908313 23616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:22.909940 23596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:50:22.910132 23596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/wal
uuid: "aef6b5782a784ac2b64e9c5c94b4033c"
format_stamp: "Formatted at 2026-08-05 22:50:22 on dist-test-slave-7lbf"
server_key: "b9c5359775cda36712565b08849a7d32"
server_key_iv: "18a69db6084c0cec2159a7ea4ee6b68b"
server_key_version: "encryptionkey@0"
I20260805 22:50:22.910336 23596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:22.930351 23596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:22.934971 23596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:22.935268 23596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:22.954447 23596 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.61:44033
I20260805 22:50:22.954515 23686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.61:44033 every 8 connection(s)
I20260805 22:50:22.955868 23596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
I20260805 22:50:22.959929 23689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:22.963385 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 23596
I20260805 22:50:22.963593 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-1/wal/instance
I20260805 22:50:22.964061 4208 external_mini_cluster.cc:1428] Setting key 93ef1fbd5fe7894d387c7122aeb05718
I20260805 22:50:22.967022 23689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:50:22 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970222, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:22.988806 23692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.969848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.4.28.62:37627)
0805 22:50:22.970368 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:22.970373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:22.971575 (+ 1202us) server_negotiation.cc:407] Connection header received
0805 22:50:22.973620 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:22.973627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:22.973722 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:22.973824 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:22.975634 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.976408 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.978159 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.978600 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.978720 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:22.982479 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:22.982485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:22.982488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:22.982541 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:22.985298 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:22.986261 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:22.986267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:22.986269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:22.986419 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:22.986901 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:22.986906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:22.986908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:22.987208 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:22.987372 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:22.988166 (+ 794us) server_negotiation.cc:299] Negotiation successful
0805 22:50:22.988426 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":297,"threads_started":1}
WARNING: no policy specified for kudu/127.4.28.60@KRBTEST.COM; defaulting to no policy
I20260805 22:50:22.989974 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.970494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37627 (local address 127.0.0.1:46600)
0805 22:50:22.971374 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.971423 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:22.973291 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:22.974125 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:22.974135 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:22.974735 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:22.975404 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:22.975424 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.976591 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.976596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:22.977990 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:22.978005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.978867 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.978876 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:22.979104 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:22.979781 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:22.979807 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:22.982182 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:22.985545 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:22.985559 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:22.985582 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:22.986114 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:22.986545 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:22.986550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:22.986555 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:22.986767 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:22.987348 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:22.987358 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:22.987758 (+ 400us) client_negotiation.cc:769] Sending connection context
0805 22:50:22.988024 (+ 266us) client_negotiation.cc:240] Negotiation successful
0805 22:50:22.988593 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":367,"threads_started":1}
I20260805 22:50:22.992055 23689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.60@KRBTEST.COM" created.
Aug 05 22:50:22 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970222, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
I20260805 22:50:23.008870 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.993012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34958)
0805 22:50:22.993211 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:22.993232 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:22.993368 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:22.994683 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:22.994688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:22.995201 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:22.995416 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:22.995429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.996896 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.996901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:22.998073 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:22.998090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.999042 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.999048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:22.999255 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:22.999923 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:22.999946 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:23.002602 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:23.006319 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.006324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.006327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.006738 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.007167 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.007171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.007173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.007331 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.008086 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:23.008090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:23.008225 (+ 135us) client_negotiation.cc:769] Sending connection context
0805 22:50:23.008360 (+ 135us) client_negotiation.cc:240] Negotiation successful
0805 22:50:23.008588 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260805 22:50:23.009279 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:22.993340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.4.28.61:44033)
0805 22:50:22.994172 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:22.994179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:22.994207 (+ 28us) server_negotiation.cc:407] Connection header received
0805 22:50:22.994257 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:22.994263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:22.994489 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:22.994624 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:22.995618 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.996724 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.998234 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:22.998854 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:22.998971 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.002867 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.002885 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.002908 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.002951 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.006147 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.006886 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.006895 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.006898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.007050 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.007446 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.007451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.007454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.007935 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.008158 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.008540 (+ 382us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.008828 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":508,"threads_started":1}
I20260805 22:50:23.011778 23689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:50:23.012916 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.012448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44020)
0805 22:50:23.012639 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.012766 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260805 22:50:23.013157 23619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
I20260805 22:50:23.014878 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.014523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44030)
0805 22:50:23.014672 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.014760 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260805 22:50:23.015108 23689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:45529: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab.
Entry for principal kudu/127.4.28.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab.
I20260805 22:50:23.029134 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.60:45529
--webserver_interface=127.4.28.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab
--principal=kudu/127.4.28.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:37627,127.4.28.61:44033,127.4.28.60:45529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:50:23.064427 23689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } attempt: 1
I20260805 22:50:23.065758 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.065248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44036)
0805 22:50:23.065465 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.065595 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260805 22:50:23.066792 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.066431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44044)
0805 22:50:23.066599 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.066674 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260805 22:50:23.067013 23689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:45529: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
I20260805 22:50:23.120886 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } attempt: 5
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970222, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
I20260805 22:50:23.135262 23689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } attempt: 2
I20260805 22:50:23.136417 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.135898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44054)
0805 22:50:23.136092 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.136204 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260805 22:50:23.136505 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.121822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.4.28.61:44033)
0805 22:50:23.122009 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.122016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.122041 (+ 25us) server_negotiation.cc:407] Connection header received
0805 22:50:23.122221 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.122225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.122314 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.122404 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.123246 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.123963 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.125195 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.125699 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.125821 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.129499 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.129508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.129533 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.129586 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.133152 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.134048 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.134055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.134059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.134228 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.134706 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.134712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.134715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.135137 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.135441 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.135936 (+ 495us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.136192 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 22:50:23.136502 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.121707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34962)
0805 22:50:23.121909 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.121931 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:23.122072 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:23.122463 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:23.122469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:23.122880 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:23.123089 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.123100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.124128 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.124133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.125030 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.125054 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.125866 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.125872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.126090 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.126782 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:23.126802 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:23.129221 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:23.133383 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.133389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.133392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.133894 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.134398 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.134403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.134406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.134577 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.135309 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:23.135315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:23.135544 (+ 229us) client_negotiation.cc:769] Sending connection context
0805 22:50:23.135766 (+ 222us) client_negotiation.cc:240] Negotiation successful
0805 22:50:23.136114 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260805 22:50:23.138144 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.137703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44070)
0805 22:50:23.137883 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.137986 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260805 22:50:23.138120 23579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } has no permanent_uuid. Determining permanent_uuid...
W20260805 22:50:23.138386 23689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:45529: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
I20260805 22:50:23.139350 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.138797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44086)
0805 22:50:23.139061 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.139158 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260805 22:50:23.140395 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.140015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44088)
0805 22:50:23.140160 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.140245 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260805 22:50:23.140646 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:45529: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
I20260805 22:50:23.191982 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } attempt: 1
I20260805 22:50:23.193408 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.192802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44096)
0805 22:50:23.193075 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.193194 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260805 22:50:23.194641 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.194233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44108)
0805 22:50:23.194398 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.194497 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260805 22:50:23.194919 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:45529: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
I20260805 22:50:23.229629 23689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } attempt: 3
I20260805 22:50:23.231009 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.230395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44124)
0805 22:50:23.230659 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.230776 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260805 22:50:23.232209 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.231682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44126)
0805 22:50:23.231960 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.232063 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260805 22:50:23.232484 23689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:45529: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
I20260805 22:50:23.263154 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } attempt: 2
I20260805 22:50:23.264698 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.263963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44140)
0805 22:50:23.264308 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.264454 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260805 22:50:23.265928 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.265495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44150)
0805 22:50:23.265711 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.265802 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260805 22:50:23.266212 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:45529: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
I20260805 22:50:23.372483 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } attempt: 3
I20260805 22:50:23.374202 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.373235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44166)
0805 22:50:23.373654 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.373942 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260805 22:50:23.375464 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.375005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44172)
0805 22:50:23.375201 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.375307 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260805 22:50:23.375700 23579 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:45529: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
I20260805 22:50:23.375691 23689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } attempt: 4
I20260805 22:50:23.376806 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.376281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44176)
0805 22:50:23.376515 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.376614 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260805 22:50:23.377920 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.377540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44178)
0805 22:50:23.377682 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.377769 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260805 22:50:23.378131 23689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:45529: Network error: Client connection negotiation failed: client connection to 127.4.28.60:45529: connect: Connection refused (error 111)
W20260805 22:50:23.397758 23699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:23.398070 23699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:23.398167 23699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:23.406149 23699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:23.406268 23699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:23.406330 23699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:23.406420 23699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:23.406482 23699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:23.417079 23699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:37627,127.4.28.61:44033,127.4.28.60:45529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab
--principal=kudu/127.4.28.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.60:45529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.28.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:23.418596 23699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:23.420040 23699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:23.432539 23708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:23.432588 23705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:23.432562 23706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:23.433326 23699 server_base.cc:1034] running on GCE node
I20260805 22:50:23.434183 23699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:23.435592 23699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:23.436782 23699 hybrid_clock.cc:648] HybridClock initialized: now 1785970223436727 us; error 50 us; skew 500 ppm
I20260805 22:50:23.437139 23699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970223, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:23.440415 23699 init.cc:377] Logged in from keytab as kudu/127.4.28.60@KRBTEST.COM (short username kudu)
I20260805 22:50:23.442065 23699 webserver.cc:460] Webserver started at http://127.4.28.60:35459/ using document root <none> and password file <none>
I20260805 22:50:23.442756 23699 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:23.442858 23699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:23.443138 23699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:23.446291 23699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/data/instance:
uuid: "1d7a4f0d8a8048d6ba92295f00472dc1"
format_stamp: "Formatted at 2026-08-05 22:50:23 on dist-test-slave-7lbf"
server_key: "fd5366d132158f657bde17fd1e1f9f62"
server_key_iv: "6370e3e759b1be3786e2b666d1837beb"
server_key_version: "encryptionkey@0"
I20260805 22:50:23.447104 23699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/wal/instance:
uuid: "1d7a4f0d8a8048d6ba92295f00472dc1"
format_stamp: "Formatted at 2026-08-05 22:50:23 on dist-test-slave-7lbf"
server_key: "fd5366d132158f657bde17fd1e1f9f62"
server_key_iv: "6370e3e759b1be3786e2b666d1837beb"
server_key_version: "encryptionkey@0"
I20260805 22:50:23.452658 23699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:50:23.456578 23717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:23.458261 23699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:50:23.458472 23699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/wal
uuid: "1d7a4f0d8a8048d6ba92295f00472dc1"
format_stamp: "Formatted at 2026-08-05 22:50:23 on dist-test-slave-7lbf"
server_key: "fd5366d132158f657bde17fd1e1f9f62"
server_key_iv: "6370e3e759b1be3786e2b666d1837beb"
server_key_version: "encryptionkey@0"
I20260805 22:50:23.458701 23699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:23.481418 23699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:23.486744 23699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:23.487057 23699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:23.507519 23699 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.60:45529
I20260805 22:50:23.507553 23786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.60:45529 every 8 connection(s)
I20260805 22:50:23.509027 23699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
I20260805 22:50:23.510043 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 23699
I20260805 22:50:23.510254 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785970002413767-4208-0/minicluster-data/master-2/wal/instance
I20260805 22:50:23.510867 4208 external_mini_cluster.cc:1428] Setting key d7794cfb183fa54f51f43dd73435b548
I20260805 22:50:23.513401 23791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:23.522431 23791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970223, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:23.541852 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.513019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.4.28.60:45529)
0805 22:50:23.518163 (+ 5144us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.518182 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.518728 (+ 546us) server_negotiation.cc:407] Connection header received
0805 22:50:23.521205 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.521240 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.521878 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.522390 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.523929 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.525722 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.527400 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.528161 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.528302 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.533479 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.533489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.533505 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.533619 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.536989 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.537827 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.537835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.537843 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.538028 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.538490 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.538494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.538497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.539159 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.539389 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.539921 (+ 532us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.540355 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4847,"thread_start_us":4574,"threads_started":1}
I20260805 22:50:23.543412 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.525748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37627 (local address 127.0.0.1:46612)
0805 22:50:23.526719 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.526773 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:23.526952 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:23.527494 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:23.527500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:23.528230 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:23.528590 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.528601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.529966 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.529972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.531079 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.531092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.532076 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.532084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.532277 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.532999 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:23.533041 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:23.535947 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:23.540229 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.540242 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.540249 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.540827 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.541275 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.541280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.541284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.541479 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.542057 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:23.542066 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:23.542337 (+ 271us) client_negotiation.cc:769] Sending connection context
0805 22:50:23.542568 (+ 231us) client_negotiation.cc:240] Negotiation successful
0805 22:50:23.542948 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":552,"threads_started":1}
I20260805 22:50:23.543403 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.525739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.4.28.62:37627)
0805 22:50:23.526294 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.526301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.526872 (+ 571us) server_negotiation.cc:407] Connection header received
0805 22:50:23.527101 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.527108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.527209 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.527329 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.528754 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.529792 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.531245 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.531796 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.531903 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.536226 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.536232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.536236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.536285 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.540077 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.540988 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.540994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.540996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.541147 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.541604 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.541609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.541612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.541911 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.542057 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.542745 (+ 688us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.542987 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":291,"threads_started":1}
I20260805 22:50:23.544934 23791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:50:23.552925 23579 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } attempt: 4
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970223, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:23.561208 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.545740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:44033 (local address 127.0.0.1:34970)
0805 22:50:23.545989 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.546006 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:23.546116 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:23.546425 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:23.546429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:23.546876 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:23.547085 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.547097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.548488 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.548492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.549931 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.549949 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.550829 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.550834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.550996 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.551585 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:23.551610 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:23.554323 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:23.558231 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.558237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.558242 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.558823 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.559360 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.559364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.559367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.559551 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.560188 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:23.560192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:23.560360 (+ 168us) client_negotiation.cc:769] Sending connection context
0805 22:50:23.560523 (+ 163us) client_negotiation.cc:240] Negotiation successful
0805 22:50:23.560842 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260805 22:50:23.561295 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.545782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.4.28.61:44033)
0805 22:50:23.545957 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.545961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.546069 (+ 108us) server_negotiation.cc:407] Connection header received
0805 22:50:23.546228 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.546232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.546292 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.546363 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.547241 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.548310 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.550079 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.550678 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.550823 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.554578 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.554584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.554587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.554623 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.558071 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.559003 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.559009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.559013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.559199 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.559697 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.559702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.559705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.560056 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.560223 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.560659 (+ 436us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.560907 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970222, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
I20260805 22:50:23.562426 23791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:50:23.563161 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.545090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.4.28.62:37627)
0805 22:50:23.545321 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.545326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.545344 (+ 18us) server_negotiation.cc:407] Connection header received
0805 22:50:23.545556 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.545562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.545642 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.545737 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.547452 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.548466 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.550335 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.551018 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.551117 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.556048 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.556053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.556057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.556089 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.559201 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.560025 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.560031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.560035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.560182 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.560663 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.560668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.560670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.561067 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.561329 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.562529 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.562759 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260805 22:50:23.569293 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.553731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.4.28.60:45529)
0805 22:50:23.553927 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.553932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.554115 (+ 183us) server_negotiation.cc:407] Connection header received
0805 22:50:23.554185 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.554190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.554271 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.554380 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.555307 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.556365 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.558281 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.558760 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.558873 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.562669 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.562676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.562680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.562717 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.566461 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.567105 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.567111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.567115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.567237 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.567658 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.567665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.567669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.568119 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.568293 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.568724 (+ 431us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.568948 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:50:23.569299 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.553633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44192)
0805 22:50:23.553931 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.553951 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:23.554074 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:23.554438 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:23.554443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:23.554918 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:23.555150 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.555165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.556530 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.556536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.558147 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.558164 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.558918 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.558928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.559104 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.559601 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:23.559622 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:23.562423 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:23.566595 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.566601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.566606 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.566974 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.567347 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.567352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.567356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.567530 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.568274 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:23.568280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:23.568458 (+ 178us) client_negotiation.cc:769] Sending connection context
0805 22:50:23.568606 (+ 148us) client_negotiation.cc:240] Negotiation successful
0805 22:50:23.568918 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
I20260805 22:50:23.576844 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6: Bootstrap starting.
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970223, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
I20260805 22:50:23.581148 23579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:23.582659 23579 log.cc:824] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:23.583513 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.565109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.4.28.61:44033)
0805 22:50:23.565286 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.565292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.565432 (+ 140us) server_negotiation.cc:407] Connection header received
0805 22:50:23.565634 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.565639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.565722 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.565824 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.567571 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.568428 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.570572 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.571301 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.571427 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.576985 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.576993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.576998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.577045 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.580038 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.581028 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.581035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.581038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.581215 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.581782 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.581786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.581788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.582097 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.582390 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.582716 (+ 326us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.582940 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 22:50:23.584533 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.563237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44198)
0805 22:50:23.563896 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.563916 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:23.564075 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:23.570103 (+ 6028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:23.570108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:23.570633 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:23.570832 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.570845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.572176 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.572181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.573560 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.573577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.574782 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.574788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.574976 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.575560 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:23.575581 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:23.578177 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:23.581712 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.581718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.581722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.582232 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.582649 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.582654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.582656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.582881 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.583543 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:23.583547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:23.583702 (+ 155us) client_negotiation.cc:769] Sending connection context
0805 22:50:23.583836 (+ 134us) client_negotiation.cc:240] Negotiation successful
0805 22:50:23.584171 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537}
I20260805 22:50:23.585276 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.563896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.4.28.60:45529)
0805 22:50:23.569690 (+ 5794us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.569696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.569712 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:50:23.569765 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.569769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.569836 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.569943 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.570985 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.572019 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.573728 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.574442 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.574554 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.578404 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.578410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.578415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.578456 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.581546 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.582370 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.582375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.582379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.582522 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.583015 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.583019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.583021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.583438 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.583845 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.583990 (+ 145us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.584967 (+ 977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":764,"server-negotiator.queue_time_us":5660,"thread_start_us":10590,"threads_started":1}
I20260805 22:50:23.586987 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6: No bootstrap required, opened a new log
I20260805 22:50:23.589460 23791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1: Bootstrap starting.
I20260805 22:50:23.593163 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:50:23.593350 23791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:23.594774 23791 log.cc:824] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:23.595568 23579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } }
I20260805 22:50:23.596029 23579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:23.596168 23579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0841c4cb4b6a472eae0a9fa57bf53fa6, State: Initialized, Role: FOLLOWER
I20260805 22:50:23.596802 23579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } }
I20260805 22:50:23.597774 23807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.598212 23807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:50:23.599022 23579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:50:23.599031 23791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1: No bootstrap required, opened a new log
I20260805 22:50:23.610841 23791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } }
I20260805 22:50:23.611351 23791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:23.611500 23791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7a4f0d8a8048d6ba92295f00472dc1, State: Initialized, Role: FOLLOWER
I20260805 22:50:23.612352 23791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } }
I20260805 22:50:23.613605 23831 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260805 22:50:23.620576 23835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.621026 23835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:50:23.623176 23791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:50:23.619315 23488 master_runner.cc:186] Error getting master registration for 127.4.28.61:44033: OK, instance_id { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" instance_seqno: 1785970222948895 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 22:50:23.630391 23488 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260805 22:50:23.631018 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.609108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.4.28.61:44033)
0805 22:50:23.609317 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.609324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.610747 (+ 1423us) server_negotiation.cc:407] Connection header received
0805 22:50:23.611545 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.611551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.611649 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.611777 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.614375 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.615490 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.618144 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.618941 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.619048 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.622969 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.622978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.622982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.623025 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.626983 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.628111 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.628119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.628124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.628320 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.628826 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.628844 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.628848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.629296 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.629594 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.630319 (+ 725us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.630650 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260805 22:50:23.636070 23520 tablet.cc:2369] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:50:23.643371 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.611018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.4.28.60:45529)
0805 22:50:23.611198 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.611204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.615272 (+ 4068us) server_negotiation.cc:407] Connection header received
0805 22:50:23.618187 (+ 2915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.618194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.618314 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.618395 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.621129 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.622246 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.630555 (+ 8309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.631076 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.631190 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.635052 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.635059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.635064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.635111 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.639143 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.640424 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.640431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.640434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.640590 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.640987 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.640992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.640995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.641346 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.641655 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.642626 (+ 971us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.642968 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260805 22:50:23.643447 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.610093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.4.28.62:37627)
0805 22:50:23.610263 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.610269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.614750 (+ 4481us) server_negotiation.cc:407] Connection header received
0805 22:50:23.615556 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.615562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.615636 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.615719 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.624448 (+ 8729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.625695 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.628581 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.629289 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.629417 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.635479 (+ 6062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.635487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.635492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.635560 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.639143 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.640562 (+ 1419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.640569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.640574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.640795 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.641324 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.641330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.641333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.641763 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.642026 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.642931 (+ 905us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.643155 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260805 22:50:23.650756 23847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 22:50:23.650935 23847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 22:50:23.652241 23699 master_runner.cc:186] Error getting master registration for 127.4.28.61:44033: OK, instance_id { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" instance_seqno: 1785970222948895 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 22:50:23.660169 23699 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260805 22:50:23.677376 23689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } attempt: 5
Aug 05 22:50:23 dist-test-slave-7lbf krb5kdc[23463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970222, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
I20260805 22:50:23.692514 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.678179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.4.28.60:45529)
0805 22:50:23.678356 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:23.678363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:23.678409 (+ 46us) server_negotiation.cc:407] Connection header received
0805 22:50:23.678617 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:23.678621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:23.678685 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:23.678769 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:23.679619 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.680385 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.681643 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.682087 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.682202 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.686412 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:23.686420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:23.686425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:23.686468 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:23.689396 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.690348 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.690353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.690356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.690518 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:23.690995 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:23.690999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:23.691002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:23.691298 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:23.691435 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:23.691977 (+ 542us) server_negotiation.cc:299] Negotiation successful
0805 22:50:23.692215 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:50:23.692524 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:23.678115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:45529 (local address 127.0.0.1:44210)
0805 22:50:23.678327 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:23.678349 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:23.678485 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:23.678814 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:23.678819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:23.679288 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:23.679461 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.679470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.680531 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.680538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.681480 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:23.681492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:23.682235 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:23.682242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:23.682443 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:23.683089 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:23.683114 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:23.686185 (+ 3071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:23.689579 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.689585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.689590 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.690175 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.690661 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:23.690667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:23.690671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:23.690856 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:23.691438 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:23.691444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:23.691629 (+ 185us) client_negotiation.cc:769] Sending connection context
0805 22:50:23.691781 (+ 152us) client_negotiation.cc:240] Negotiation successful
0805 22:50:23.692113 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260805 22:50:23.698246 23689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c: Bootstrap starting.
I20260805 22:50:23.701692 23689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:23.702884 23689 log.cc:824] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:23.705819 23689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c: No bootstrap required, opened a new log
I20260805 22:50:23.714170 23689 raft_consensus.cc:348] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } }
I20260805 22:50:23.714540 23689 raft_consensus.cc:374] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:23.714658 23689 raft_consensus.cc:729] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef6b5782a784ac2b64e9c5c94b4033c, State: Initialized, Role: FOLLOWER
I20260805 22:50:23.715296 23689 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } }
I20260805 22:50:23.716312 23850 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.716738 23850 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:50:23.717692 23689 sys_catalog.cc:564] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [sys.catalog]: configured and running, proceeding with master startup.
W20260805 22:50:23.727780 23867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 22:50:23.727941 23867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 22:50:23.729974 23850 raft_consensus.cc:482] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:50:23.730299 23850 raft_consensus.cc:504] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } }
I20260805 22:50:23.731940 23850 leader_election.cc:288] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0841c4cb4b6a472eae0a9fa57bf53fa6 (127.4.28.62:37627), 1d7a4f0d8a8048d6ba92295f00472dc1 (127.4.28.60:45529)
I20260805 22:50:23.732836 23748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" is_pre_election: true
I20260805 22:50:23.733563 23748 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aef6b5782a784ac2b64e9c5c94b4033c in term 0.
I20260805 22:50:23.734445 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" is_pre_election: true
I20260805 22:50:23.735069 23546 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aef6b5782a784ac2b64e9c5c94b4033c in term 0.
I20260805 22:50:23.735267 23619 leader_election.cc:302] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d7a4f0d8a8048d6ba92295f00472dc1, aef6b5782a784ac2b64e9c5c94b4033c; no voters:
I20260805 22:50:23.735751 23850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:50:23.735916 23850 raft_consensus.cc:482] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:50:23.736021 23850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:23.738034 23596 master_runner.cc:263] No leader master found from master aef6b5782a784ac2b64e9c5c94b4033c
I20260805 22:50:23.738196 23596 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260805 22:50:23.739403 23723 tablet.cc:2369] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:50:23.739799 23850 raft_consensus.cc:504] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } }
I20260805 22:50:23.741855 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6"
I20260805 22:50:23.742092 23546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:23.745002 23748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1"
I20260805 22:50:23.745265 23748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:23.745548 23546 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aef6b5782a784ac2b64e9c5c94b4033c in term 1.
I20260805 22:50:23.746040 23617 leader_election.cc:302] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0841c4cb4b6a472eae0a9fa57bf53fa6, aef6b5782a784ac2b64e9c5c94b4033c; no voters:
I20260805 22:50:23.749063 23748 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aef6b5782a784ac2b64e9c5c94b4033c in term 1.
I20260805 22:50:23.749994 23850 leader_election.cc:288] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [CANDIDATE]: Term 1 election: Requested vote from peers 0841c4cb4b6a472eae0a9fa57bf53fa6 (127.4.28.62:37627), 1d7a4f0d8a8048d6ba92295f00472dc1 (127.4.28.60:45529)
I20260805 22:50:23.750336 23850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:50:23.750603 23850 raft_consensus.cc:686] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [term 1 LEADER]: Becoming Leader. State: Replica: aef6b5782a784ac2b64e9c5c94b4033c, State: Running, Role: LEADER
I20260805 22:50:23.751334 23850 consensus_queue.cc:237] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } }
I20260805 22:50:23.764262 23871 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef6b5782a784ac2b64e9c5c94b4033c. Latest consensus state: current_term: 1 leader_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.764842 23871 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:23.765940 23875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:50:23.768513 23875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:50:23.781570 23748 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [term 1 FOLLOWER]: Refusing update from remote peer aef6b5782a784ac2b64e9c5c94b4033c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:23.782809 23871 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:23.783396 23546 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [term 1 FOLLOWER]: Refusing update from remote peer aef6b5782a784ac2b64e9c5c94b4033c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:23.784417 23871 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:23.808455 23877 mvcc.cc:205] Tried to move back new op lower bound from 7315334036587122688 to 7315334036492771328. Current Snapshot: MvccSnapshot[applied={T|T < 7315334036587122688}]
I20260805 22:50:23.810052 23871 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.810386 23871 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:23.810997 23835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef6b5782a784ac2b64e9c5c94b4033c. Latest consensus state: current_term: 1 leader_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.811651 23835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:50:23.812310 23807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef6b5782a784ac2b64e9c5c94b4033c. Latest consensus state: current_term: 1 leader_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.812639 23807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:50:23.813366 23878 mvcc.cc:205] Tried to move back new op lower bound from 7315334036587122688 to 7315334036492771328. Current Snapshot: MvccSnapshot[applied={T|T < 7315334036587122688}]
I20260805 22:50:23.814010 23875 catalog_manager.cc:1347] Generated new cluster ID: 337cfa1722bc4d48bfb4e82109123d88
I20260805 22:50:23.814128 23875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:50:23.826917 23850 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.827301 23850 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:23.829020 23835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.829306 23835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:50:23.830263 23807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0841c4cb4b6a472eae0a9fa57bf53fa6" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37627 } } peers { permanent_uuid: "aef6b5782a784ac2b64e9c5c94b4033c" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 44033 } } peers { permanent_uuid: "1d7a4f0d8a8048d6ba92295f00472dc1" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 45529 } } }
I20260805 22:50:23.830567 23807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:50:23.835315 23875 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:50:23.837239 23875 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:50:23.851816 23875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aef6b5782a784ac2b64e9c5c94b4033c: Generated new TSK 0
I20260805 22:50:23.853077 23875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:50:24.618567 23831 catalog_manager.cc:1259] Loaded cluster ID: 337cfa1722bc4d48bfb4e82109123d88
I20260805 22:50:24.618695 23831 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6: loading cluster ID for follower catalog manager: success
I20260805 22:50:24.621779 23831 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6: acquiring CA information for follower catalog manager: success
I20260805 22:50:24.623971 23831 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0841c4cb4b6a472eae0a9fa57bf53fa6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 22:50:24.653102 23847 catalog_manager.cc:1259] Loaded cluster ID: 337cfa1722bc4d48bfb4e82109123d88
I20260805 22:50:24.653223 23847 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1: loading cluster ID for follower catalog manager: success
I20260805 22:50:24.655973 23847 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1: acquiring CA information for follower catalog manager: success
I20260805 22:50:24.658113 23847 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d7a4f0d8a8048d6ba92295f00472dc1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 22:50:24.953715 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 23488
I20260805 22:50:24.955124 23571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:25.075614 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 23488
I20260805 22:50:25.103268 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 23596
I20260805 22:50:25.104637 23680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:25.216823 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 23596
W20260805 22:50:25.245586 23719 connection.cc:535] server connection from 127.0.0.1:44210 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260805 22:50:25.245914 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 23699
I20260805 22:50:25.247242 23778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:25.357705 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 23699
2026-08-05T22:50:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4984 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:50:25 dist-test-slave-7lbf krb5kdc[23912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:50:25 dist-test-slave-7lbf krb5kdc[23912](info): set up 2 sockets
Aug 05 22:50:25 dist-test-slave-7lbf krb5kdc[23912](info): commencing operation
krb5kdc: starting...
W20260805 22:50:26.585119 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:50:26 dist-test-slave-7lbf krb5kdc[23912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:50:26.904935 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:32979
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:32979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:27.268438 23933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:27.268781 23933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:27.268887 23933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:27.276588 23933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:27.276721 23933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:27.276767 23933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:27.276832 23933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:27.276877 23933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:27.287451 23933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:32979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:27.288890 23933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:27.290342 23933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:27.302452 23942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:27.302510 23945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:27.302843 23943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:27.304054 23933 server_base.cc:1034] running on GCE node
I20260805 22:50:27.305035 23933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:27.306682 23933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:27.307910 23933 hybrid_clock.cc:648] HybridClock initialized: now 1785970227307853 us; error 45 us; skew 500 ppm
I20260805 22:50:27.308310 23933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:27 dist-test-slave-7lbf krb5kdc[23912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970227, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:27.311671 23933 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:50:27.313341 23933 webserver.cc:460] Webserver started at http://127.4.28.62:44403/ using document root <none> and password file <none>
I20260805 22:50:27.314129 23933 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:27.314261 23933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:27.314585 23933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:27.317912 23933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "3626daadda2840da878c31f4fcec9466"
format_stamp: "Formatted at 2026-08-05 22:50:27 on dist-test-slave-7lbf"
server_key: "16825a015938c95697d34a8f911315c3"
server_key_iv: "ec3198da06c63e8afb722fe4ce580a1f"
server_key_version: "encryptionkey@0"
I20260805 22:50:27.318734 23933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "3626daadda2840da878c31f4fcec9466"
format_stamp: "Formatted at 2026-08-05 22:50:27 on dist-test-slave-7lbf"
server_key: "16825a015938c95697d34a8f911315c3"
server_key_iv: "ec3198da06c63e8afb722fe4ce580a1f"
server_key_version: "encryptionkey@0"
I20260805 22:50:27.324878 23933 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260805 22:50:27.328878 23955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:27.330499 23933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260805 22:50:27.330726 23933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "3626daadda2840da878c31f4fcec9466"
format_stamp: "Formatted at 2026-08-05 22:50:27 on dist-test-slave-7lbf"
server_key: "16825a015938c95697d34a8f911315c3"
server_key_iv: "ec3198da06c63e8afb722fe4ce580a1f"
server_key_version: "encryptionkey@0"
I20260805 22:50:27.330991 23933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:27.363103 23933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:27.367600 23933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:27.367916 23933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:27.387229 23933 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:32979
I20260805 22:50:27.387248 24035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:32979 every 8 connection(s)
I20260805 22:50:27.388760 23933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:50:27.392999 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:27.396836 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 23933
I20260805 22:50:27.397089 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:50:27.397894 4208 external_mini_cluster.cc:1428] Setting key 3ca8702b7312e37cbdf960a5bb393fe9
I20260805 22:50:27.406540 24039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466: Bootstrap starting.
I20260805 22:50:27.410612 24039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:27.411954 24039 log.cc:824] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:27.415516 24039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466: No bootstrap required, opened a new log
Aug 05 22:50:27 dist-test-slave-7lbf krb5kdc[23912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:27.425949 24039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3626daadda2840da878c31f4fcec9466" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 32979 } }
I20260805 22:50:27.426491 24039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:27.426605 24039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3626daadda2840da878c31f4fcec9466, State: Initialized, Role: FOLLOWER
I20260805 22:50:27.427073 24043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:27.400190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.4.28.62:32979)
0805 22:50:27.401099 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:27.401116 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:27.401174 (+ 58us) server_negotiation.cc:407] Connection header received
0805 22:50:27.404514 (+ 3340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:27.404557 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:27.405228 (+ 671us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:27.405788 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:27.407342 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:27.409430 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:27.411220 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:27.412321 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:27.412539 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:27.417380 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:27.417405 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:27.417426 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:27.417485 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:27.421635 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:27.422565 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:27.422576 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:27.422587 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:27.422846 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:27.423324 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:27.423329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:27.423333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:27.424196 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:27.424535 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:27.424953 (+ 418us) server_negotiation.cc:299] Negotiation successful
0805 22:50:27.425460 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":368,"threads_started":1}
I20260805 22:50:27.427376 24039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3626daadda2840da878c31f4fcec9466" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 32979 } }
I20260805 22:50:27.427666 24039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:50:27.427851 24039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:50:27.428040 24039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:27.430758 24039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3626daadda2840da878c31f4fcec9466" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 32979 } }
I20260805 22:50:27.431299 24039 leader_election.cc:302] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3626daadda2840da878c31f4fcec9466; no voters:
I20260805 22:50:27.431764 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:50:27.432014 24039 leader_election.cc:288] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:50:27.432157 24045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:50:27.434125 24039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:50:27.435276 24045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [term 1 LEADER]: Becoming Leader. State: Replica: 3626daadda2840da878c31f4fcec9466, State: Running, Role: LEADER
I20260805 22:50:27.435940 24045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3626daadda2840da878c31f4fcec9466" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 32979 } }
I20260805 22:50:27.439083 24047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3626daadda2840da878c31f4fcec9466. Latest consensus state: current_term: 1 leader_uuid: "3626daadda2840da878c31f4fcec9466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3626daadda2840da878c31f4fcec9466" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 32979 } } }
I20260805 22:50:27.439564 24047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:27.440138 24046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3626daadda2840da878c31f4fcec9466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3626daadda2840da878c31f4fcec9466" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 32979 } } }
I20260805 22:50:27.440689 24046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:27.441581 24056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:50:27.447932 24056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:50:27.460137 24056 catalog_manager.cc:1347] Generated new cluster ID: 5c447e97bf2e4db2a73982606b9d9e87
I20260805 22:50:27.460279 24056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:50:27.478327 24056 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:50:27.480391 24056 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:50:27.500303 24056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3626daadda2840da878c31f4fcec9466: Generated new TSK 0
I20260805 22:50:27.501573 24056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:50:27.546415 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 23933
I20260805 22:50:27.547741 24030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:27.646926 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 23933
2026-08-05T22:50:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2290 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:50:27 dist-test-slave-7lbf krb5kdc[24084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:50:27 dist-test-slave-7lbf krb5kdc[24084](info): set up 2 sockets
Aug 05 22:50:27 dist-test-slave-7lbf krb5kdc[24084](info): commencing operation
krb5kdc: starting...
W20260805 22:50:28.868322 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.004s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:50:29 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:50:29.189705 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45197
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:29.551312 24102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:29.551659 24102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:29.551805 24102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:29.559342 24102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:29.559494 24102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:29.559579 24102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:29.559660 24102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:29.559723 24102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:29.570169 24102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:29.571698 24102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:29.573180 24102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:29.585589 24108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:29.585537 24109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:29.585748 24111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:29.586303 24102 server_base.cc:1034] running on GCE node
I20260805 22:50:29.587136 24102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:29.588766 24102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:29.590011 24102 hybrid_clock.cc:648] HybridClock initialized: now 1785970229589952 us; error 40 us; skew 500 ppm
I20260805 22:50:29.590384 24102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:29 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970229, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:29.593653 24102 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:50:29.596274 24102 webserver.cc:460] Webserver started at http://127.4.28.62:36299/ using document root <none> and password file <none>
I20260805 22:50:29.597195 24102 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:29.597363 24102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:29.597826 24102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:29.601178 24102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "209edec2d6ff49a994e44042d0c1ac62"
format_stamp: "Formatted at 2026-08-05 22:50:29 on dist-test-slave-7lbf"
server_key: "3f53d53afa123cff754f448a0c62c3c5"
server_key_iv: "bd2de377b4a42fa76dfc795560567885"
server_key_version: "encryptionkey@0"
I20260805 22:50:29.602088 24102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "209edec2d6ff49a994e44042d0c1ac62"
format_stamp: "Formatted at 2026-08-05 22:50:29 on dist-test-slave-7lbf"
server_key: "3f53d53afa123cff754f448a0c62c3c5"
server_key_iv: "bd2de377b4a42fa76dfc795560567885"
server_key_version: "encryptionkey@0"
I20260805 22:50:29.607947 24102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260805 22:50:29.612207 24123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:29.613888 24102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:50:29.614095 24102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "209edec2d6ff49a994e44042d0c1ac62"
format_stamp: "Formatted at 2026-08-05 22:50:29 on dist-test-slave-7lbf"
server_key: "3f53d53afa123cff754f448a0c62c3c5"
server_key_iv: "bd2de377b4a42fa76dfc795560567885"
server_key_version: "encryptionkey@0"
I20260805 22:50:29.614332 24102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:29.635032 24102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:29.639721 24102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:29.640079 24102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:29.660533 24102 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:45197
I20260805 22:50:29.660557 24209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:45197 every 8 connection(s)
I20260805 22:50:29.662272 24102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:50:29.662721 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 24102
I20260805 22:50:29.662928 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:50:29.663708 4208 external_mini_cluster.cc:1428] Setting key 1579ff10d03816d55f656ea02648e9ef
I20260805 22:50:29.667282 24210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:50:29 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:29.682251 24210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Bootstrap starting.
I20260805 22:50:29.686218 24210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:29.687471 24210 log.cc:824] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:29.689407 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:29.666122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.4.28.62:45197)
0805 22:50:29.666946 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:29.666958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:29.667004 (+ 46us) server_negotiation.cc:407] Connection header received
0805 22:50:29.668711 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:29.668736 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:29.669161 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:29.669571 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:29.671695 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:29.673133 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:29.675195 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:29.675915 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:29.676058 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:29.681356 (+ 5298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:29.681366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:29.681385 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:29.681434 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:29.684582 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:29.685396 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:29.685405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:29.685412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:29.685667 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:29.686096 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:29.686106 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:29.686109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:29.686745 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:29.687063 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:29.687572 (+ 509us) server_negotiation.cc:299] Negotiation successful
0805 22:50:29.687999 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":360,"threads_started":1}
I20260805 22:50:29.692000 24210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: No bootstrap required, opened a new log
I20260805 22:50:29.700513 24210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:29.701016 24210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:29.701119 24210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209edec2d6ff49a994e44042d0c1ac62, State: Initialized, Role: FOLLOWER
I20260805 22:50:29.701773 24210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:29.702049 24210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:50:29.702165 24210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:50:29.702318 24210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:29.704859 24210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:29.705415 24210 leader_election.cc:302] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 209edec2d6ff49a994e44042d0c1ac62; no voters:
I20260805 22:50:29.706156 24210 leader_election.cc:288] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:50:29.706293 24215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:50:29.706986 24215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 1 LEADER]: Becoming Leader. State: Replica: 209edec2d6ff49a994e44042d0c1ac62, State: Running, Role: LEADER
I20260805 22:50:29.707571 24215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:29.708518 24210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:50:29.711156 24216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "209edec2d6ff49a994e44042d0c1ac62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } } }
I20260805 22:50:29.711647 24216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:29.712612 24223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:50:29.712718 24218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209edec2d6ff49a994e44042d0c1ac62. Latest consensus state: current_term: 1 leader_uuid: "209edec2d6ff49a994e44042d0c1ac62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } } }
I20260805 22:50:29.713052 24218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:29.719295 24223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:50:29.729568 24223 catalog_manager.cc:1347] Generated new cluster ID: e04a850d9e02480892fb6ee2b4076e13
I20260805 22:50:29.729705 24223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:50:29.740844 24223 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:50:29.742944 24223 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:50:29.755693 24223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Generated new TSK 0
I20260805 22:50:29.756809 24223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:50:29.846724 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45197
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:30.220592 24243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:30.220916 24243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:30.221033 24243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:30.228621 24243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:30.228763 24243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:30.228883 24243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:50:30.239255 24243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:30.240677 24243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:30.242141 24243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:30.256647 24250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:30.256583 24251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:30.256665 24255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:30.262957 24243 server_base.cc:1034] running on GCE node
I20260805 22:50:30.263741 24243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:30.264873 24243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:30.266108 24243 hybrid_clock.cc:648] HybridClock initialized: now 1785970230266056 us; error 43 us; skew 500 ppm
I20260805 22:50:30.266448 24243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:30 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:30.269953 24243 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:50:30.271641 24243 webserver.cc:460] Webserver started at http://127.4.28.1:40103/ using document root <none> and password file <none>
I20260805 22:50:30.272449 24243 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:30.272578 24243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:30.272908 24243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:30.276162 24243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "1bb030ffcc794c31ac1176184e65b8cc"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "53567f229e58666cbfb01f1e52d02c2a"
server_key_iv: "6c4e622490ffcfc03e019a7d39a93432"
server_key_version: "encryptionkey@0"
I20260805 22:50:30.277004 24243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "1bb030ffcc794c31ac1176184e65b8cc"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "53567f229e58666cbfb01f1e52d02c2a"
server_key_iv: "6c4e622490ffcfc03e019a7d39a93432"
server_key_version: "encryptionkey@0"
I20260805 22:50:30.282858 24243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260805 22:50:30.286782 24269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:30.288290 24243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:50:30.288465 24243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "1bb030ffcc794c31ac1176184e65b8cc"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "53567f229e58666cbfb01f1e52d02c2a"
server_key_iv: "6c4e622490ffcfc03e019a7d39a93432"
server_key_version: "encryptionkey@0"
I20260805 22:50:30.288673 24243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:30.312999 24243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:30.317705 24243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:30.318044 24243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:30.319205 24243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:30.320761 24243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:30.320844 24243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:30.320950 24243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:30.321008 24243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:30.355232 24243 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:33755
I20260805 22:50:30.355266 24423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:33755 every 8 connection(s)
I20260805 22:50:30.356717 24243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:50:30.365572 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 24243
I20260805 22:50:30.365826 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:50:30.366698 4208 external_mini_cluster.cc:1428] Setting key 797c5508b4724c46959a353478fa0600
Aug 05 22:50:30 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:30.379488 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:30.359594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:59119 (local address 127.4.28.62:45197)
0805 22:50:30.360100 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:30.360107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:30.361196 (+ 1089us) server_negotiation.cc:407] Connection header received
0805 22:50:30.363237 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:30.363242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:30.363324 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:30.363438 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:30.365259 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:30.366447 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:30.368526 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:30.369100 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:30.369233 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:30.373259 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:30.373265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:30.373269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:30.373309 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:30.375969 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:30.376897 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:30.376902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:30.376904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:30.377059 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:30.377556 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:30.377561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:30.377563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:30.377861 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:30.378019 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:30.378838 (+ 819us) server_negotiation.cc:299] Negotiation successful
0805 22:50:30.379114 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
I20260805 22:50:30.380570 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:30.360166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:59119)
0805 22:50:30.361001 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:30.361057 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:30.362877 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:30.363728 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:30.363738 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:30.364360 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:30.365020 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:30.365041 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:30.366670 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:30.366674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:30.368333 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:30.368348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:30.369363 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:30.369372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:30.369675 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:30.370370 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:30.370396 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:30.372964 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:30.376197 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:30.376208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:30.376225 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:30.376749 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:30.377177 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:30.377182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:30.377186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:30.377407 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:30.378003 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:30.378013 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:30.378407 (+ 394us) client_negotiation.cc:769] Sending connection context
0805 22:50:30.378699 (+ 292us) client_negotiation.cc:240] Negotiation successful
0805 22:50:30.379298 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":297,"threads_started":1}
I20260805 22:50:30.382318 24424 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45197
I20260805 22:50:30.382737 24424 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:30.383524 24424 heartbeater.cc:511] Master 127.4.28.62:45197 requested a full tablet report, sending...
I20260805 22:50:30.385807 24150 ts_manager.cc:194] Registered new tserver with Master: 1bb030ffcc794c31ac1176184e65b8cc (127.4.28.1:33755)
I20260805 22:50:30.387372 24150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:59119
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:50:30.435359 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45197
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:30.802994 24431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:30.803313 24431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:30.803422 24431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:30.811369 24431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:30.811503 24431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:30.811635 24431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:50:30.822314 24431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:30.823753 24431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:30.825125 24431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:30.838974 24444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:30.839088 24445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:30.839113 24449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:30.839756 24431 server_base.cc:1034] running on GCE node
I20260805 22:50:30.840462 24431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:30.841574 24431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:30.842808 24431 hybrid_clock.cc:648] HybridClock initialized: now 1785970230842740 us; error 62 us; skew 500 ppm
I20260805 22:50:30.843154 24431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:30 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:30.846547 24431 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:50:30.848173 24431 webserver.cc:460] Webserver started at http://127.4.28.2:46633/ using document root <none> and password file <none>
I20260805 22:50:30.848941 24431 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:30.849072 24431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:30.849393 24431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:30.852622 24431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "92359575832c4c0bb4340ab51f4ec106"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "9589e19b3d226bd07530ea78c918bbd8"
server_key_iv: "4825ba912ebb390a632bfd3b78858953"
server_key_version: "encryptionkey@0"
I20260805 22:50:30.853443 24431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "92359575832c4c0bb4340ab51f4ec106"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "9589e19b3d226bd07530ea78c918bbd8"
server_key_iv: "4825ba912ebb390a632bfd3b78858953"
server_key_version: "encryptionkey@0"
I20260805 22:50:30.859095 24431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260805 22:50:30.863159 24459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:30.864773 24431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:50:30.864974 24431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "92359575832c4c0bb4340ab51f4ec106"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "9589e19b3d226bd07530ea78c918bbd8"
server_key_iv: "4825ba912ebb390a632bfd3b78858953"
server_key_version: "encryptionkey@0"
I20260805 22:50:30.865195 24431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:30.898504 24431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:30.903400 24431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:30.903734 24431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:30.904835 24431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:30.906466 24431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:30.906555 24431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:30.906687 24431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:30.906751 24431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:30.940125 24431 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:36177
I20260805 22:50:30.940222 24617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:36177 every 8 connection(s)
I20260805 22:50:30.941552 24431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:50:30.948983 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 24431
I20260805 22:50:30.949179 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:50:30.949703 4208 external_mini_cluster.cc:1428] Setting key bfa3cbb1170841fa5f1ac052e33291f2
Aug 05 22:50:30 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:30.964879 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:30.944289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:37045 (local address 127.4.28.62:45197)
0805 22:50:30.944728 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:30.944733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:30.945919 (+ 1186us) server_negotiation.cc:407] Connection header received
0805 22:50:30.947903 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:30.947909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:30.947999 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:30.948104 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:30.949932 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:30.950936 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:30.953075 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:30.953681 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:30.953818 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:30.957879 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:30.957886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:30.957890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:30.957938 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:30.961180 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:30.962167 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:30.962173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:30.962176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:30.962348 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:30.962807 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:30.962811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:30.962814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:30.963185 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:30.963333 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:30.964275 (+ 942us) server_negotiation.cc:299] Negotiation successful
0805 22:50:30.964520 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":207,"threads_started":1}
I20260805 22:50:30.966097 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:30.944941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:37045)
0805 22:50:30.945729 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:30.945773 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:30.947583 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:30.948386 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:30.948396 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:30.948980 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:30.949711 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:30.949734 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:30.951151 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:30.951156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:30.952885 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:30.952901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:30.953992 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:30.954003 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:30.954235 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:30.955008 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:30.955034 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:30.957537 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:30.961386 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:30.961396 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:30.961414 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:30.962001 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:30.962461 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:30.962465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:30.962467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:30.962692 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:30.963361 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:30.963368 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:30.963779 (+ 411us) client_negotiation.cc:769] Sending connection context
0805 22:50:30.964137 (+ 358us) client_negotiation.cc:240] Negotiation successful
0805 22:50:30.964602 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":291,"threads_started":1}
I20260805 22:50:30.967878 24618 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45197
I20260805 22:50:30.968289 24618 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:30.969395 24618 heartbeater.cc:511] Master 127.4.28.62:45197 requested a full tablet report, sending...
I20260805 22:50:30.971583 24150 ts_manager.cc:194] Registered new tserver with Master: 92359575832c4c0bb4340ab51f4ec106 (127.4.28.2:36177)
I20260805 22:50:30.972828 24150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:37045
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:50:31.024614 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45197
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:31.388499 24630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:31.388880 24630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:31.388990 24630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260805 22:50:31.390803 24424 heartbeater.cc:503] Master 127.4.28.62:45197 was elected leader, sending a full tablet report...
W20260805 22:50:31.398022 24630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:31.398169 24630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:31.398303 24630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:50:31.409126 24630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:31.410799 24630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:31.412335 24630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:31.427806 24639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:31.427927 24637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:31.427946 24636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:31.429171 24630 server_base.cc:1034] running on GCE node
I20260805 22:50:31.429953 24630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:31.431106 24630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:31.432375 24630 hybrid_clock.cc:648] HybridClock initialized: now 1785970231432297 us; error 62 us; skew 500 ppm
I20260805 22:50:31.432735 24630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:31 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970231, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:31.436615 24630 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:50:31.438544 24630 webserver.cc:460] Webserver started at http://127.4.28.3:46729/ using document root <none> and password file <none>
I20260805 22:50:31.439330 24630 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:31.439457 24630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:31.439778 24630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:31.443226 24630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "8d6cfd11a6ee4807b67f4f103e9631fd"
format_stamp: "Formatted at 2026-08-05 22:50:31 on dist-test-slave-7lbf"
server_key: "e8c7cdc7c43937669b6933d4908e08cc"
server_key_iv: "e035a3227781f03940189323554fe7c2"
server_key_version: "encryptionkey@0"
I20260805 22:50:31.444072 24630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "8d6cfd11a6ee4807b67f4f103e9631fd"
format_stamp: "Formatted at 2026-08-05 22:50:31 on dist-test-slave-7lbf"
server_key: "e8c7cdc7c43937669b6933d4908e08cc"
server_key_iv: "e035a3227781f03940189323554fe7c2"
server_key_version: "encryptionkey@0"
I20260805 22:50:31.450448 24630 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260805 22:50:31.454762 24647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:31.456542 24630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260805 22:50:31.456733 24630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "8d6cfd11a6ee4807b67f4f103e9631fd"
format_stamp: "Formatted at 2026-08-05 22:50:31 on dist-test-slave-7lbf"
server_key: "e8c7cdc7c43937669b6933d4908e08cc"
server_key_iv: "e035a3227781f03940189323554fe7c2"
server_key_version: "encryptionkey@0"
I20260805 22:50:31.456955 24630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:31.488059 24630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:31.492651 24630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:31.492952 24630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:31.494091 24630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:31.495585 24630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:31.495667 24630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:31.495757 24630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:31.495806 24630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:31.528926 24630 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:44223
I20260805 22:50:31.529024 24812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:44223 every 8 connection(s)
I20260805 22:50:31.530296 24630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:50:31.537931 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 24630
I20260805 22:50:31.538122 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:50:31.538552 4208 external_mini_cluster.cc:1428] Setting key c2ede7edee131d4cb14319febaa422e6
Aug 05 22:50:31 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970231, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:31.552325 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.533086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:56033 (local address 127.4.28.62:45197)
0805 22:50:31.533665 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:31.533671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:31.534690 (+ 1019us) server_negotiation.cc:407] Connection header received
0805 22:50:31.536672 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:31.536677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:31.536756 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:31.536861 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:31.538645 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.539697 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.541793 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.542377 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.542502 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.546464 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:31.546470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:31.546474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:31.546516 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:31.549217 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:31.550099 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:31.550104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:31.550107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:31.550238 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:31.550655 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:31.550658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:31.550659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:31.550928 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:31.551049 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:31.551811 (+ 762us) server_negotiation.cc:299] Negotiation successful
0805 22:50:31.552047 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":257,"threads_started":1}
I20260805 22:50:31.553367 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.533677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:56033)
0805 22:50:31.534500 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:31.534548 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:31.536332 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:31.537133 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:31.537142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:31.537757 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:31.538434 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.538455 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.539903 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.539909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.541550 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.541570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.542649 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.542655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.542900 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.543605 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:31.543630 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:31.546154 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:31.549407 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:31.549416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:31.549433 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:31.549974 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:31.550333 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:31.550336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:31.550338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:31.550569 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:31.551063 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:31.551071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:31.551441 (+ 370us) client_negotiation.cc:769] Sending connection context
0805 22:50:31.551684 (+ 243us) client_negotiation.cc:240] Negotiation successful
0805 22:50:31.552140 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":306,"threads_started":1}
I20260805 22:50:31.554993 24813 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45197
I20260805 22:50:31.555333 24813 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:31.556074 24813 heartbeater.cc:511] Master 127.4.28.62:45197 requested a full tablet report, sending...
I20260805 22:50:31.557674 24150 ts_manager.cc:194] Registered new tserver with Master: 8d6cfd11a6ee4807b67f4f103e9631fd (127.4.28.3:44223)
I20260805 22:50:31.558523 24150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:56033
I20260805 22:50:31.567785 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:50:31.586381 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.572706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.4.28.62:45197)
0805 22:50:31.572885 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:31.572891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:31.573313 (+ 422us) server_negotiation.cc:407] Connection header received
0805 22:50:31.573468 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:31.573472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:31.573573 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:31.573658 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:31.575146 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.575953 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.577361 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.577894 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.578003 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.580774 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:31.580779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:31.580783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:31.580855 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:31.583449 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:31.584245 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:31.584250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:31.584252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:31.584421 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:31.584864 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:31.584869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:31.584871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:31.585236 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:31.585434 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:31.585900 (+ 466us) server_negotiation.cc:299] Negotiation successful
0805 22:50:31.586115 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 22:50:31.590735 24150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:50:31.598140 24150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:50:31.626643 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.612405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33755 (local address 127.0.0.1:45996)
0805 22:50:31.614613 (+ 2208us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:31.614636 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:31.614805 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:31.616053 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:31.616065 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:31.616115 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:31.616708 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.616725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.619193 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.619201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.622215 (+ 3014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.622242 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.625169 (+ 2927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.625177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.625486 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.625586 (+ 100us) client_negotiation.cc:769] Sending connection context
0805 22:50:31.625816 (+ 230us) client_negotiation.cc:240] Negotiation successful
0805 22:50:31.626096 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"thread_start_us":1859,"threads_started":1}
I20260805 22:50:31.628127 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.613098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.4.28.1:33755)
0805 22:50:31.614091 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:31.614103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:31.614962 (+ 859us) server_negotiation.cc:407] Connection header received
0805 22:50:31.615111 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:31.615122 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:31.615431 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:31.615690 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:31.617013 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.618940 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.622519 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.624454 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.624627 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.624972 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:31.627521 (+ 2549us) server_negotiation.cc:299] Negotiation successful
0805 22:50:31.627728 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":545,"threads_started":1}
I20260805 22:50:31.632647 24337 tablet_service.cc:1467] Processing CreateTablet for tablet 27142877043142fc82ebc0203b1f4684 (DEFAULT_TABLE table=test-table [id=8c632ad73bb949469abd120bf2e9ff1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:31.634538 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27142877043142fc82ebc0203b1f4684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:31.635972 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.613809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:36177 (local address 127.0.0.1:41986)
0805 22:50:31.626337 (+ 12528us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:31.626360 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:31.626511 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:31.628730 (+ 2219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:31.628735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:31.628768 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:31.629232 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.629248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.631451 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.631461 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.633370 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.633384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.634895 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.634903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.635157 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.635200 (+ 43us) client_negotiation.cc:769] Sending connection context
0805 22:50:31.635288 (+ 88us) client_negotiation.cc:240] Negotiation successful
0805 22:50:31.635437 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12360,"mutex_wait_us":40,"thread_start_us":12810,"threads_started":1}
I20260805 22:50:31.636080 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.611706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:44223 (local address 127.0.0.1:58140)
0805 22:50:31.612850 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:31.612916 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:31.613133 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:31.628236 (+ 15103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:31.628241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:31.628273 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:31.628647 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.628663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.630879 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.630885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.632909 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.632926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.635064 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.635072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.635331 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.635373 (+ 42us) client_negotiation.cc:769] Sending connection context
0805 22:50:31.635456 (+ 83us) client_negotiation.cc:240] Negotiation successful
0805 22:50:31.635615 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":1196,"threads_started":1}
I20260805 22:50:31.637131 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.614378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.4.28.2:36177)
0805 22:50:31.627876 (+ 13498us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:31.627882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:31.627908 (+ 26us) server_negotiation.cc:407] Connection header received
0805 22:50:31.627996 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:31.628002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:31.628195 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:31.628715 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:31.629481 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.631288 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.633594 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.634622 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.636393 (+ 1771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.636620 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:31.636711 (+ 91us) server_negotiation.cc:299] Negotiation successful
0805 22:50:31.636886 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13334,"thread_start_us":13199,"threads_started":1}
I20260805 22:50:31.637367 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.612115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.4.28.3:44223)
0805 22:50:31.627463 (+ 15348us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:31.627473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:31.627515 (+ 42us) server_negotiation.cc:407] Connection header received
0805 22:50:31.627641 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:31.627650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:31.627977 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:31.628254 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:31.629041 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.630724 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.633114 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.634746 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.636382 (+ 1636us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.636666 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:31.636759 (+ 93us) server_negotiation.cc:299] Negotiation successful
0805 22:50:31.636956 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15155,"thread_start_us":14972,"threads_started":1}
I20260805 22:50:31.641296 24717 tablet_service.cc:1467] Processing CreateTablet for tablet 27142877043142fc82ebc0203b1f4684 (DEFAULT_TABLE table=test-table [id=8c632ad73bb949469abd120bf2e9ff1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:31.641296 24523 tablet_service.cc:1467] Processing CreateTablet for tablet 27142877043142fc82ebc0203b1f4684 (DEFAULT_TABLE table=test-table [id=8c632ad73bb949469abd120bf2e9ff1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:31.643026 24717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27142877043142fc82ebc0203b1f4684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:31.643024 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27142877043142fc82ebc0203b1f4684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:31.648571 24840 tablet_bootstrap.cc:492] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: Bootstrap starting.
I20260805 22:50:31.652004 24840 tablet_bootstrap.cc:654] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:31.654036 24840 log.cc:824] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:31.659292 24842 tablet_bootstrap.cc:492] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: Bootstrap starting.
I20260805 22:50:31.660766 24840 tablet_bootstrap.cc:492] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: No bootstrap required, opened a new log
I20260805 22:50:31.661072 24840 ts_tablet_manager.cc:1397] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260805 22:50:31.662050 24844 tablet_bootstrap.cc:492] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: Bootstrap starting.
I20260805 22:50:31.664253 24842 tablet_bootstrap.cc:654] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:31.665835 24842 log.cc:824] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:31.666251 24844 tablet_bootstrap.cc:654] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:31.667825 24844 log.cc:824] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:31.671994 24840 raft_consensus.cc:348] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } }
I20260805 22:50:31.672523 24840 raft_consensus.cc:374] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:31.672621 24840 raft_consensus.cc:729] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb030ffcc794c31ac1176184e65b8cc, State: Initialized, Role: FOLLOWER
I20260805 22:50:31.673287 24840 consensus_queue.cc:260] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } }
I20260805 22:50:31.676298 24842 tablet_bootstrap.cc:492] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: No bootstrap required, opened a new log
I20260805 22:50:31.676687 24842 ts_tablet_manager.cc:1397] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.013s
I20260805 22:50:31.683568 24840 ts_tablet_manager.cc:1428] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: Time spent starting tablet: real 0.022s user 0.018s sys 0.003s
I20260805 22:50:31.685026 24844 tablet_bootstrap.cc:492] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: No bootstrap required, opened a new log
I20260805 22:50:31.685360 24844 ts_tablet_manager.cc:1397] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.012s
I20260805 22:50:31.690171 24842 raft_consensus.cc:348] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } }
I20260805 22:50:31.690675 24842 raft_consensus.cc:374] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:31.690780 24842 raft_consensus.cc:729] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92359575832c4c0bb4340ab51f4ec106, State: Initialized, Role: FOLLOWER
I20260805 22:50:31.691368 24842 consensus_queue.cc:260] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } }
I20260805 22:50:31.692617 24618 heartbeater.cc:503] Master 127.4.28.62:45197 was elected leader, sending a full tablet report...
I20260805 22:50:31.693251 24842 ts_tablet_manager.cc:1428] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260805 22:50:31.693661 24621 tablet.cc:2369] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:50:31.695003 24844 raft_consensus.cc:348] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } }
I20260805 22:50:31.695410 24844 raft_consensus.cc:374] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:31.695515 24844 raft_consensus.cc:729] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6cfd11a6ee4807b67f4f103e9631fd, State: Initialized, Role: FOLLOWER
I20260805 22:50:31.696135 24844 consensus_queue.cc:260] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } }
I20260805 22:50:31.697817 24813 heartbeater.cc:503] Master 127.4.28.62:45197 was elected leader, sending a full tablet report...
I20260805 22:50:31.698295 24844 ts_tablet_manager.cc:1428] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260805 22:50:31.782083 24815 tablet.cc:2369] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:50:31.789767 24852 raft_consensus.cc:482] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:50:31.790084 24852 raft_consensus.cc:504] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } }
I20260805 22:50:31.791817 24852 leader_election.cc:288] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92359575832c4c0bb4340ab51f4ec106 (127.4.28.2:36177), 1bb030ffcc794c31ac1176184e65b8cc (127.4.28.1:33755)
I20260805 22:50:31.798715 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.792349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:36177 (local address 127.4.28.3:34025)
0805 22:50:31.792619 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:31.792640 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:31.792788 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:31.793304 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:31.793309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:31.793345 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:31.793619 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.793633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.794950 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.794955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.796730 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.796747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.798099 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.798104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.798264 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.798292 (+ 28us) client_negotiation.cc:769] Sending connection context
0805 22:50:31.798372 (+ 80us) client_negotiation.cc:240] Negotiation successful
0805 22:50:31.798476 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":1}
I20260805 22:50:31.798820 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.792349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:34025 (local address 127.4.28.2:36177)
0805 22:50:31.792586 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:31.792592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:31.792743 (+ 151us) server_negotiation.cc:407] Connection header received
0805 22:50:31.792951 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:31.792958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:31.793054 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:31.793163 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:31.793832 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.794781 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.796927 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.797930 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.798079 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.798184 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:31.798419 (+ 235us) server_negotiation.cc:299] Negotiation successful
0805 22:50:31.798558 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 22:50:31.799911 24853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.792349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33755 (local address 127.4.28.3:57805)
0805 22:50:31.793623 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:31.793643 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:31.793780 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:31.794152 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:31.794156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:31.794180 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:31.794532 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.794543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.795691 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.795697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.797425 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:31.797438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.799207 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.799214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:31.799407 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.799443 (+ 36us) client_negotiation.cc:769] Sending connection context
0805 22:50:31.799522 (+ 79us) client_negotiation.cc:240] Negotiation successful
0805 22:50:31.799618 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":269,"thread_start_us":440,"threads_started":1}
I20260805 22:50:31.799991 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.792447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:57805 (local address 127.4.28.1:33755)
0805 22:50:31.792647 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:31.792652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:31.793776 (+ 1124us) server_negotiation.cc:407] Connection header received
0805 22:50:31.793907 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:31.793913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:31.793996 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:31.794102 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:31.794713 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.795520 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.797617 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.798823 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.798963 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.799065 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:31.799611 (+ 546us) server_negotiation.cc:299] Negotiation successful
0805 22:50:31.799754 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260805 22:50:31.800041 24552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27142877043142fc82ebc0203b1f4684" candidate_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92359575832c4c0bb4340ab51f4ec106" is_pre_election: true
I20260805 22:50:31.800675 24552 raft_consensus.cc:2454] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6cfd11a6ee4807b67f4f103e9631fd in term 0.
I20260805 22:50:31.800946 24363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27142877043142fc82ebc0203b1f4684" candidate_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb030ffcc794c31ac1176184e65b8cc" is_pre_election: true
I20260805 22:50:31.801496 24363 raft_consensus.cc:2454] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6cfd11a6ee4807b67f4f103e9631fd in term 0.
I20260805 22:50:31.801496 24648 leader_election.cc:302] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6cfd11a6ee4807b67f4f103e9631fd, 92359575832c4c0bb4340ab51f4ec106; no voters:
I20260805 22:50:31.801916 24852 raft_consensus.cc:2781] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:50:31.802096 24852 raft_consensus.cc:482] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:50:31.802218 24852 raft_consensus.cc:3037] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:31.804469 24852 raft_consensus.cc:504] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } }
I20260805 22:50:31.805088 24852 leader_election.cc:288] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [CANDIDATE]: Term 1 election: Requested vote from peers 92359575832c4c0bb4340ab51f4ec106 (127.4.28.2:36177), 1bb030ffcc794c31ac1176184e65b8cc (127.4.28.1:33755)
I20260805 22:50:31.805667 24552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27142877043142fc82ebc0203b1f4684" candidate_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92359575832c4c0bb4340ab51f4ec106"
I20260805 22:50:31.805713 24363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27142877043142fc82ebc0203b1f4684" candidate_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb030ffcc794c31ac1176184e65b8cc"
I20260805 22:50:31.805872 24552 raft_consensus.cc:3037] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:31.805912 24363 raft_consensus.cc:3037] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:31.808583 24552 raft_consensus.cc:2454] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6cfd11a6ee4807b67f4f103e9631fd in term 1.
I20260805 22:50:31.808634 24363 raft_consensus.cc:2454] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6cfd11a6ee4807b67f4f103e9631fd in term 1.
I20260805 22:50:31.809056 24648 leader_election.cc:302] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6cfd11a6ee4807b67f4f103e9631fd, 92359575832c4c0bb4340ab51f4ec106; no voters:
I20260805 22:50:31.809423 24852 raft_consensus.cc:2781] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:50:31.809929 24852 raft_consensus.cc:686] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 1 LEADER]: Becoming Leader. State: Replica: 8d6cfd11a6ee4807b67f4f103e9631fd, State: Running, Role: LEADER
I20260805 22:50:31.810456 24852 consensus_queue.cc:237] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } }
I20260805 22:50:31.815747 24150 catalog_manager.cc:5495] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d6cfd11a6ee4807b67f4f103e9631fd (127.4.28.3). New cstate: current_term: 1 leader_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } health_report { overall_health: UNKNOWN } } }
W20260805 22:50:31.859555 24425 tablet.cc:2369] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:50:31.867646 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.861849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.4.28.3:44223)
0805 22:50:31.862051 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:31.862056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:31.862181 (+ 125us) server_negotiation.cc:407] Connection header received
0805 22:50:31.862392 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:31.862396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:31.862487 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:31.862638 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:50:31.862986 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.863717 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.865293 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.865787 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.865949 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.866510 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:50:31.867117 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:50:31.867236 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:31.867343 (+ 107us) server_negotiation.cc:299] Negotiation successful
0805 22:50:31.867425 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 22:50:31.876567 24363 raft_consensus.cc:1261] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [term 1 FOLLOWER]: Refusing update from remote peer 8d6cfd11a6ee4807b67f4f103e9631fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:31.876567 24552 raft_consensus.cc:1261] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [term 1 FOLLOWER]: Refusing update from remote peer 8d6cfd11a6ee4807b67f4f103e9631fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:31.877594 24855 consensus_queue.cc:1035] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:31.878043 24852 consensus_queue.cc:1035] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:31.890476 24861 mvcc.cc:205] Tried to move back new op lower bound from 7315334069756063744 to 7315334069499564032. Current Snapshot: MvccSnapshot[applied={T|T < 7315334069756063744}]
I20260805 22:50:31.895561 24858 mvcc.cc:205] Tried to move back new op lower bound from 7315334069756063744 to 7315334069499564032. Current Snapshot: MvccSnapshot[applied={T|T < 7315334069756063744 or (T in {7315334069756063744})}]
I20260805 22:50:31.908524 24150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:50:31.908864 24150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:50:31.912716 24150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Sending DeleteTablet for 3 replicas of tablet 27142877043142fc82ebc0203b1f4684
I20260805 22:50:31.914176 24717 tablet_service.cc:1514] Processing DeleteTablet for tablet 27142877043142fc82ebc0203b1f4684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:31 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:58140
I20260805 22:50:31.914258 24337 tablet_service.cc:1514] Processing DeleteTablet for tablet 27142877043142fc82ebc0203b1f4684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:31 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:45996
I20260805 22:50:31.914610 24523 tablet_service.cc:1514] Processing DeleteTablet for tablet 27142877043142fc82ebc0203b1f4684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:31 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:41986
I20260805 22:50:31.915557 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 24243
I20260805 22:50:31.940845 24874 tablet_replica.cc:330] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: stopping tablet replica
I20260805 22:50:31.941422 24874 raft_consensus.cc:2229] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:50:31.942240 24874 raft_consensus.cc:2258] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:31.943579 24873 tablet_replica.cc:330] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: stopping tablet replica
I20260805 22:50:31.944093 24873 raft_consensus.cc:2229] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:50:31.944711 24873 raft_consensus.cc:2258] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:31.945443 24875 tablet_replica.cc:330] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: stopping tablet replica
I20260805 22:50:31.945901 24875 raft_consensus.cc:2229] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:50:31.946497 24875 raft_consensus.cc:2258] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:31.949302 24873 ts_tablet_manager.cc:1905] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:31.949294 24874 ts_tablet_manager.cc:1905] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:31.957410 24874 ts_tablet_manager.cc:1918] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:31.957707 24874 log.cc:1196] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/27142877043142fc82ebc0203b1f4684
I20260805 22:50:31.958280 24874 ts_tablet_manager.cc:1939] T 27142877043142fc82ebc0203b1f4684 P 8d6cfd11a6ee4807b67f4f103e9631fd: Deleting consensus metadata
I20260805 22:50:31.959118 24875 ts_tablet_manager.cc:1905] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:31.960449 24124 catalog_manager.cc:4841] TS 8d6cfd11a6ee4807b67f4f103e9631fd (127.4.28.3:44223): tablet 27142877043142fc82ebc0203b1f4684 (table test-table [id=8c632ad73bb949469abd120bf2e9ff1f]) successfully deleted
I20260805 22:50:31.960747 24873 ts_tablet_manager.cc:1918] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:31.961585 24873 log.cc:1196] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/27142877043142fc82ebc0203b1f4684
I20260805 22:50:31.962455 24873 ts_tablet_manager.cc:1939] T 27142877043142fc82ebc0203b1f4684 P 1bb030ffcc794c31ac1176184e65b8cc: Deleting consensus metadata
Aug 05 22:50:31 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:50:31.964778 24129 catalog_manager.cc:4841] TS 1bb030ffcc794c31ac1176184e65b8cc (127.4.28.1:33755): tablet 27142877043142fc82ebc0203b1f4684 (table test-table [id=8c632ad73bb949469abd120bf2e9ff1f]) successfully deleted
I20260805 22:50:31.967512 24875 ts_tablet_manager.cc:1918] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:31.967746 24875 log.cc:1196] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/27142877043142fc82ebc0203b1f4684
I20260805 22:50:31.968320 24875 ts_tablet_manager.cc:1939] T 27142877043142fc82ebc0203b1f4684 P 92359575832c4c0bb4340ab51f4ec106: Deleting consensus metadata
I20260805 22:50:31.973994 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:31.916770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.4.28.1:33755)
0805 22:50:31.916988 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:31.916996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:31.939250 (+ 22254us) server_negotiation.cc:407] Connection header received
0805 22:50:31.945652 (+ 6402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:31.945658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:31.945756 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:31.946277 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:31.948263 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.949384 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.951629 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:31.952302 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:31.952409 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:31.965896 (+ 13487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:31.965914 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:31.965934 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:31.965997 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:31.970115 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:31.971243 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:31.971254 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:31.971258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:31.971449 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:31.971937 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:31.971942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:31.971946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:31.972413 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:31.972705 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:31.973331 (+ 626us) server_negotiation.cc:299] Negotiation successful
0805 22:50:31.973611 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260805 22:50:31.975473 24417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:31.976728 24124 catalog_manager.cc:4841] TS 92359575832c4c0bb4340ab51f4ec106 (127.4.28.2:36177): tablet 27142877043142fc82ebc0203b1f4684 (table test-table [id=8c632ad73bb949469abd120bf2e9ff1f]) successfully deleted
I20260805 22:50:32.083824 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 24243
I20260805 22:50:32.107710 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 24431
Aug 05 22:50:32 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:50:32.126562 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:32.108649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.4.28.2:36177)
0805 22:50:32.108902 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:32.108909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:32.108933 (+ 24us) server_negotiation.cc:407] Connection header received
0805 22:50:32.109075 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:32.109081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:32.109191 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:32.109305 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:32.110766 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:32.111939 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:32.113311 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:32.113976 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:32.114117 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:32.118885 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:32.118911 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:32.118934 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:32.118993 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:32.123142 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:32.124087 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:32.124099 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:32.124102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:32.124293 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:32.124709 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:32.124714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:32.124718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:32.125267 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:32.125660 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:32.125918 (+ 258us) server_negotiation.cc:299] Negotiation successful
0805 22:50:32.126170 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260805 22:50:32.127436 24609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:32.219735 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 24431
I20260805 22:50:32.252132 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 24630
Aug 05 22:50:32 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:50:32.271867 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:32.253666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.4.28.3:44223)
0805 22:50:32.253853 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:32.253859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:32.253878 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:50:32.253953 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:32.253958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:32.254053 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:32.254159 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:32.255533 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:32.256601 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:32.258315 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:32.259176 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:32.259306 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:32.265037 (+ 5731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:32.265058 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:32.265076 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:32.265132 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:32.268406 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:32.269287 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:32.269297 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:32.269300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:32.269499 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:32.270035 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:32.270039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:32.270041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:32.270462 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:32.270838 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:32.271228 (+ 390us) server_negotiation.cc:299] Negotiation successful
0805 22:50:32.271474 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 22:50:32.272713 24807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:32.401381 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 24630
I20260805 22:50:32.430474 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 24102
I20260805 22:50:32.431692 24201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:32.545032 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 24102
I20260805 22:50:32.577448 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45197
--webserver_interface=127.4.28.62
--webserver_port=36299
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:32.954183 24882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:32.954526 24882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:32.954638 24882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:32.962230 24882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:32.962368 24882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:32.962414 24882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:32.962468 24882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:32.962512 24882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:32.973042 24882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:32.974501 24882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:32.975869 24882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:33.002812 24890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:33.002851 24893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:33.002754 24891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:33.003219 24882 server_base.cc:1034] running on GCE node
I20260805 22:50:33.004469 24882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:33.006310 24882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:33.007575 24882 hybrid_clock.cc:648] HybridClock initialized: now 1785970233007488 us; error 50 us; skew 500 ppm
I20260805 22:50:33.008142 24882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:33 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970233, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:33.016754 24882 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:50:33.018884 24882 webserver.cc:460] Webserver started at http://127.4.28.62:36299/ using document root <none> and password file <none>
I20260805 22:50:33.019773 24882 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:33.019948 24882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:33.027132 24882 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260805 22:50:33.030732 24905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:33.032655 24882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260805 22:50:33.032896 24882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "209edec2d6ff49a994e44042d0c1ac62"
format_stamp: "Formatted at 2026-08-05 22:50:29 on dist-test-slave-7lbf"
server_key: "3f53d53afa123cff754f448a0c62c3c5"
server_key_iv: "bd2de377b4a42fa76dfc795560567885"
server_key_version: "encryptionkey@0"
I20260805 22:50:33.034123 24882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:33.057816 24882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:33.062731 24882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:33.063050 24882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:33.083178 24882 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:45197
I20260805 22:50:33.083200 24976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:45197 every 8 connection(s)
I20260805 22:50:33.084856 24882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:50:33.090358 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 24882
I20260805 22:50:33.091522 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:33755
--local_ip_for_outbound_sockets=127.4.28.1
--tserver_master_addrs=127.4.28.62:45197
--webserver_port=40103
--webserver_interface=127.4.28.1
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:50:33.093902 24977 sys_catalog.cc:263] Verifying existing consensus state
I20260805 22:50:33.099057 24977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Bootstrap starting.
I20260805 22:50:33.122565 24977 log.cc:824] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:33.136649 24977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:50:33.137259 24977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Bootstrap complete.
I20260805 22:50:33.147466 24977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:33.148375 24977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209edec2d6ff49a994e44042d0c1ac62, State: Initialized, Role: FOLLOWER
I20260805 22:50:33.149080 24977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:33.149317 24977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:50:33.149444 24977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:50:33.149644 24977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:50:33.153932 24977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:33.154520 24977 leader_election.cc:302] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 209edec2d6ff49a994e44042d0c1ac62; no voters:
I20260805 22:50:33.155180 24977 leader_election.cc:288] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260805 22:50:33.155329 24982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:50:33.156558 24982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 2 LEADER]: Becoming Leader. State: Replica: 209edec2d6ff49a994e44042d0c1ac62, State: Running, Role: LEADER
I20260805 22:50:33.157153 24982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:33.157727 24977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:50:33.159876 24983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "209edec2d6ff49a994e44042d0c1ac62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } } }
I20260805 22:50:33.160305 24983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:33.164163 24984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209edec2d6ff49a994e44042d0c1ac62. Latest consensus state: current_term: 2 leader_uuid: "209edec2d6ff49a994e44042d0c1ac62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } } }
I20260805 22:50:33.164551 24984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:33.166519 24992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:50:33.176126 24992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:50:33.178051 24992 catalog_manager.cc:1259] Loaded cluster ID: e04a850d9e02480892fb6ee2b4076e13
I20260805 22:50:33.178187 24992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:50:33.182341 24992 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260805 22:50:33.203178 24992 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260805 22:50:33.203858 25008 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260805 22:50:33.211320 24992 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:50:33.212884 24992 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Loaded TSK: 0
I20260805 22:50:33.214202 24992 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 22:50:33.495887 24980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:33.496243 24980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:33.496390 24980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:33.504184 24980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:33.504328 24980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:33.504488 24980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:50:33.515084 24980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:33755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=40103
--tserver_master_addrs=127.4.28.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:33.516526 24980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:33.518085 24980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:33.534161 25026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:33.534724 25018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:33.534787 25019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:33.535418 24980 server_base.cc:1034] running on GCE node
I20260805 22:50:33.536253 24980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:33.537587 24980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:33.538861 24980 hybrid_clock.cc:648] HybridClock initialized: now 1785970233538767 us; error 75 us; skew 500 ppm
I20260805 22:50:33.539315 24980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:33 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970233, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:33.543594 24980 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:50:33.545492 24980 webserver.cc:460] Webserver started at http://127.4.28.1:40103/ using document root <none> and password file <none>
I20260805 22:50:33.546360 24980 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:33.546499 24980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:33.552870 24980 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260805 22:50:33.556113 25035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:33.557844 24980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 22:50:33.558040 24980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "1bb030ffcc794c31ac1176184e65b8cc"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "53567f229e58666cbfb01f1e52d02c2a"
server_key_iv: "6c4e622490ffcfc03e019a7d39a93432"
server_key_version: "encryptionkey@0"
I20260805 22:50:33.559195 24980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:33.580538 24980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:33.585165 24980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:33.585440 24980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:33.586563 24980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:33.588048 24980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:33.588147 24980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:33.588258 24980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:33.588335 24980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:33.620658 24980 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:33755
I20260805 22:50:33.620754 25200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:33755 every 8 connection(s)
I20260805 22:50:33.622202 24980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:50:33.626307 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 24980
I20260805 22:50:33.626994 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:36177
--local_ip_for_outbound_sockets=127.4.28.2
--tserver_master_addrs=127.4.28.62:45197
--webserver_port=46633
--webserver_interface=127.4.28.2
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:50:33 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970233, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:33.649442 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:33.625273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:44823 (local address 127.4.28.62:45197)
0805 22:50:33.625966 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:33.625984 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:33.626763 (+ 779us) server_negotiation.cc:407] Connection header received
0805 22:50:33.629141 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:33.629161 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:33.629720 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:33.630176 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:33.631909 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:33.633904 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:33.635681 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:33.636686 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:33.636888 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:33.641027 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:33.641036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:33.641055 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:33.641103 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:33.644232 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:33.645175 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:33.645183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:33.645191 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:33.645388 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:33.645942 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:33.645948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:33.645951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:33.646518 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:33.646708 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:33.647645 (+ 937us) server_negotiation.cc:299] Negotiation successful
0805 22:50:33.648083 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":304,"threads_started":1}
I20260805 22:50:33.649427 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:33.625599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:44823)
0805 22:50:33.626536 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:33.626595 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:33.628451 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:33.630375 (+ 1924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:33.630386 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:33.631001 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:33.631612 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:33.631633 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:33.634129 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:33.634134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:33.635484 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:33.635502 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:33.636943 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:33.636949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:33.637164 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:33.637973 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:33.638007 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:33.640674 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:33.644457 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:33.644466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:33.644480 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:33.644996 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:33.645580 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:33.645585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:33.645589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:33.645811 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:33.646691 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:33.646700 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:33.647079 (+ 379us) client_negotiation.cc:769] Sending connection context
0805 22:50:33.647344 (+ 265us) client_negotiation.cc:240] Negotiation successful
0805 22:50:33.648010 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":373,"threads_started":1}
I20260805 22:50:33.652733 25201 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45197
I20260805 22:50:33.653062 25201 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:33.653900 25201 heartbeater.cc:511] Master 127.4.28.62:45197 requested a full tablet report, sending...
I20260805 22:50:33.656322 24928 ts_manager.cc:194] Registered new tserver with Master: 1bb030ffcc794c31ac1176184e65b8cc (127.4.28.1:33755)
I20260805 22:50:33.658001 24928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:44823
W20260805 22:50:33.988441 25209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:33.988780 25209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:33.988909 25209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:33.996850 25209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:33.997031 25209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:33.997179 25209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:50:34.008123 25209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:36177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=46633
--tserver_master_addrs=127.4.28.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:34.009649 25209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:34.011137 25209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:34.025082 25218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:34.025108 25220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:34.025211 25216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:34.025990 25209 server_base.cc:1034] running on GCE node
I20260805 22:50:34.026672 25209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:34.027660 25209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:34.028913 25209 hybrid_clock.cc:648] HybridClock initialized: now 1785970234028837 us; error 78 us; skew 500 ppm
I20260805 22:50:34.029238 25209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:34 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:34.032486 25209 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:50:34.034160 25209 webserver.cc:460] Webserver started at http://127.4.28.2:46633/ using document root <none> and password file <none>
I20260805 22:50:34.034904 25209 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:34.035036 25209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:34.040846 25209 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 22:50:34.043820 25235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:34.045271 25209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260805 22:50:34.045447 25209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "92359575832c4c0bb4340ab51f4ec106"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "9589e19b3d226bd07530ea78c918bbd8"
server_key_iv: "4825ba912ebb390a632bfd3b78858953"
server_key_version: "encryptionkey@0"
I20260805 22:50:34.046550 25209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:34.063975 25209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:34.068594 25209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:34.068863 25209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:34.070004 25209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:34.071537 25209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:34.071641 25209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:34.071733 25209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:34.071810 25209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:34.106518 25209 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:36177
I20260805 22:50:34.106591 25412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:36177 every 8 connection(s)
I20260805 22:50:34.107999 25209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:50:34.110818 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 25209
I20260805 22:50:34.111563 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:44223
--local_ip_for_outbound_sockets=127.4.28.3
--tserver_master_addrs=127.4.28.62:45197
--webserver_port=46729
--webserver_interface=127.4.28.3
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:50:34 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:34.129937 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.110795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:39587 (local address 127.4.28.62:45197)
0805 22:50:34.110994 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.111001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.112532 (+ 1531us) server_negotiation.cc:407] Connection header received
0805 22:50:34.114485 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.114492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.114614 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.114707 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:34.116349 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.117082 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.118798 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.119248 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.119369 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.123357 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:34.123363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:34.123367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:34.123406 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:34.126293 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:34.127290 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:34.127295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:34.127297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:34.127459 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:34.128009 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:34.128013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:34.128015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:34.128312 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:34.128469 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.129258 (+ 789us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.129591 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260805 22:50:34.131004 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.111502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:39587)
0805 22:50:34.112343 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:34.112400 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:34.114179 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:34.114985 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:34.114993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:34.115536 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:34.116153 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.116172 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.117254 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.117259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.118633 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.118648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.119514 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.119523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.119758 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.120450 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:34.120476 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:34.123044 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:34.126485 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:34.126495 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:34.126515 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:34.127136 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:34.127609 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:34.127613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:34.127616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:34.127848 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:34.128455 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:34.128464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:34.128846 (+ 382us) client_negotiation.cc:769] Sending connection context
0805 22:50:34.129113 (+ 267us) client_negotiation.cc:240] Negotiation successful
0805 22:50:34.129739 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":352,"threads_started":1}
I20260805 22:50:34.132813 25413 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45197
I20260805 22:50:34.133170 25413 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:34.134045 25413 heartbeater.cc:511] Master 127.4.28.62:45197 requested a full tablet report, sending...
I20260805 22:50:34.135752 24928 ts_manager.cc:194] Registered new tserver with Master: 92359575832c4c0bb4340ab51f4ec106 (127.4.28.2:36177)
I20260805 22:50:34.136600 24928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:39587
W20260805 22:50:34.476613 25416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:34.477008 25416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:34.477154 25416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:34.484685 25416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:34.484838 25416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:34.484997 25416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:50:34.495453 25416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:44223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=46729
--tserver_master_addrs=127.4.28.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:34.496919 25416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:34.498438 25416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:34.513044 25421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:34.512945 25426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:34.513005 25422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:34.513903 25416 server_base.cc:1034] running on GCE node
I20260805 22:50:34.514611 25416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:34.515791 25416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:34.517014 25416 hybrid_clock.cc:648] HybridClock initialized: now 1785970234516956 us; error 56 us; skew 500 ppm
I20260805 22:50:34.517356 25416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:34 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:34.520854 25416 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:50:34.522596 25416 webserver.cc:460] Webserver started at http://127.4.28.3:46729/ using document root <none> and password file <none>
I20260805 22:50:34.523383 25416 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:34.523533 25416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:34.529618 25416 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260805 22:50:34.532781 25434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:34.534512 25416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260805 22:50:34.534713 25416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "8d6cfd11a6ee4807b67f4f103e9631fd"
format_stamp: "Formatted at 2026-08-05 22:50:31 on dist-test-slave-7lbf"
server_key: "e8c7cdc7c43937669b6933d4908e08cc"
server_key_iv: "e035a3227781f03940189323554fe7c2"
server_key_version: "encryptionkey@0"
I20260805 22:50:34.535810 25416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:34.582094 25416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:34.586796 25416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:34.587118 25416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:34.588212 25416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:34.589778 25416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:34.589898 25416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:34.590013 25416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:34.590086 25416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:34.623834 25416 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:44223
I20260805 22:50:34.623883 25616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:44223 every 8 connection(s)
I20260805 22:50:34.625350 25416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:50:34.626605 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 25416
Aug 05 22:50:34 dist-test-slave-7lbf krb5kdc[24084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:34.644229 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.627738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.4.28.62:45197)
0805 22:50:34.627972 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.627980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.628413 (+ 433us) server_negotiation.cc:407] Connection header received
0805 22:50:34.628652 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.628656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.628747 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.628879 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:34.631136 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.632121 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.633576 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.634065 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.634205 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.637510 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:34.637563 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:34.637568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:34.637617 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:34.640883 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:34.641944 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:34.641949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:34.641951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:34.642094 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:34.642577 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:34.642581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:34.642583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:34.642910 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:34.643202 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.643591 (+ 389us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.643901 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":73}
I20260805 22:50:34.651775 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.628944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:59163 (local address 127.4.28.62:45197)
0805 22:50:34.629423 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.629427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.631920 (+ 2493us) server_negotiation.cc:407] Connection header received
0805 22:50:34.633942 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.633947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.634033 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.634155 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:34.636627 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.637446 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.639418 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.640153 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.640314 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.644527 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:34.644534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:34.644538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:34.644580 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:34.647995 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:34.648899 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:34.648903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:34.648905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:34.649093 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:34.649735 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:34.649742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:34.649746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:34.650178 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:34.650307 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.651216 (+ 909us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.651447 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":390,"thread_start_us":271,"threads_started":1}
I20260805 22:50:34.652870 25624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.628916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:59163)
0805 22:50:34.631733 (+ 2817us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:34.631788 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:34.633658 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:34.634507 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:34.634520 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:34.635392 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:34.636370 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.636401 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.637691 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.637696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.639252 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.639268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.640500 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.640511 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.640790 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.641534 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:34.641573 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:34.644270 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:34.648209 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:34.648219 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:34.648236 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:34.648770 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:34.649239 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:34.649244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:34.649248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:34.649582 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:34.650304 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:34.650311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:34.650720 (+ 409us) client_negotiation.cc:769] Sending connection context
0805 22:50:34.651056 (+ 336us) client_negotiation.cc:240] Negotiation successful
0805 22:50:34.651571 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2499,"thread_start_us":2254,"threads_started":1}
I20260805 22:50:34.654763 25619 heartbeater.cc:347] Connected to a master server at 127.4.28.62:45197
I20260805 22:50:34.655131 25619 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:34.656215 25619 heartbeater.cc:511] Master 127.4.28.62:45197 requested a full tablet report, sending...
I20260805 22:50:34.658100 24928 ts_manager.cc:194] Registered new tserver with Master: 8d6cfd11a6ee4807b67f4f103e9631fd (127.4.28.3:44223)
I20260805 22:50:34.659075 24928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:59163
I20260805 22:50:34.661550 25201 heartbeater.cc:503] Master 127.4.28.62:45197 was elected leader, sending a full tablet report...
I20260805 22:50:34.662406 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:50:34.671070 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.663520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.4.28.62:45197)
0805 22:50:34.663758 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.663762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.664272 (+ 510us) server_negotiation.cc:407] Connection header received
0805 22:50:34.664544 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.664549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.664702 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.664890 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:50:34.665412 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.666265 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.668130 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.668769 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.668967 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.669748 (+ 781us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:50:34.670421 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:50:34.670574 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.670723 (+ 149us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.670812 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260805 22:50:34.672138 24928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:50:34.679813 24928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:50:34.706648 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.694044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.4.28.1:33755)
0805 22:50:34.696924 (+ 2880us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.696932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.697679 (+ 747us) server_negotiation.cc:407] Connection header received
0805 22:50:34.697794 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.697801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.698112 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.698326 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:34.699362 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.701120 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.704061 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.705232 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.705394 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.705699 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.706029 (+ 330us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.706216 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2701,"thread_start_us":2557,"threads_started":1}
I20260805 22:50:34.707366 25632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.693367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33755 (local address 127.0.0.1:46012)
0805 22:50:34.697347 (+ 3980us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:34.697366 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:34.697507 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:34.698636 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:34.698642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:34.698674 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:34.699140 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.699153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.701334 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.701340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.703880 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.703894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.705487 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.705494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.705758 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.705815 (+ 57us) client_negotiation.cc:769] Sending connection context
0805 22:50:34.705938 (+ 123us) client_negotiation.cc:240] Negotiation successful
0805 22:50:34.706160 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3853,"mutex_wait_us":48,"thread_start_us":946,"threads_started":1}
I20260805 22:50:34.708930 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.693571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.4.28.3:44223)
0805 22:50:34.696544 (+ 2973us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.696554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.696597 (+ 43us) server_negotiation.cc:407] Connection header received
0805 22:50:34.696717 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.696726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.697029 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.697252 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:34.698169 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.699752 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.703807 (+ 4055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.705300 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.705445 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.705720 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.708339 (+ 2619us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.708561 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2771,"thread_start_us":2597,"threads_started":1}
I20260805 22:50:34.709568 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.693244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:44223 (local address 127.0.0.1:58182)
0805 22:50:34.694806 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:34.694858 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:34.695104 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:34.697313 (+ 2209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:34.697323 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:34.697374 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:34.697886 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.697902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.699954 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.699960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.703568 (+ 3608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.703587 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.705877 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.705884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.708045 (+ 2161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.708113 (+ 68us) client_negotiation.cc:769] Sending connection context
0805 22:50:34.708906 (+ 793us) client_negotiation.cc:240] Negotiation successful
0805 22:50:34.709089 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"mutex_wait_us":729,"thread_start_us":3818,"threads_started":1}
I20260805 22:50:34.712924 25110 tablet_service.cc:1467] Processing CreateTablet for tablet 37c912d82055460aa8defcebfd0d3d17 (DEFAULT_TABLE table=test-table [id=010e2855f7094745a59346041edcd70c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:34.714031 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.697778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:36177 (local address 127.0.0.1:42004)
0805 22:50:34.703927 (+ 6149us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:34.703947 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:34.704078 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:34.704915 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:34.704920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:34.704952 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:34.705335 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.705349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.708728 (+ 3379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.708760 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.710938 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.710956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.713151 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.713158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.713350 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.713385 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:50:34.713489 (+ 104us) client_negotiation.cc:240] Negotiation successful
0805 22:50:34.713670 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6004,"thread_start_us":5815,"threads_started":1}
I20260805 22:50:34.714145 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.697844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.4.28.2:36177)
0805 22:50:34.702631 (+ 4787us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.702641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.704220 (+ 1579us) server_negotiation.cc:407] Connection header received
0805 22:50:34.704346 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.704355 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.704659 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.704895 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:34.705599 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.708518 (+ 2919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.711132 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.712612 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.712754 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.712975 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.713614 (+ 639us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.713804 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4574,"thread_start_us":4800,"threads_started":1}
I20260805 22:50:34.713750 25515 tablet_service.cc:1467] Processing CreateTablet for tablet 37c912d82055460aa8defcebfd0d3d17 (DEFAULT_TABLE table=test-table [id=010e2855f7094745a59346041edcd70c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:34.714671 25110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c912d82055460aa8defcebfd0d3d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:34.715301 25515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c912d82055460aa8defcebfd0d3d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:34.718225 25306 tablet_service.cc:1467] Processing CreateTablet for tablet 37c912d82055460aa8defcebfd0d3d17 (DEFAULT_TABLE table=test-table [id=010e2855f7094745a59346041edcd70c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:34.719784 25306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c912d82055460aa8defcebfd0d3d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:34.728330 25640 tablet_bootstrap.cc:492] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: Bootstrap starting.
I20260805 22:50:34.730693 25643 tablet_bootstrap.cc:492] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: Bootstrap starting.
I20260805 22:50:34.732321 25640 tablet_bootstrap.cc:654] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:34.733789 25643 tablet_bootstrap.cc:654] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:34.733839 25640 log.cc:824] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:34.735029 25643 log.cc:824] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:34.737615 25640 tablet_bootstrap.cc:492] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: No bootstrap required, opened a new log
I20260805 22:50:34.738055 25640 ts_tablet_manager.cc:1397] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260805 22:50:34.738343 25647 tablet_bootstrap.cc:492] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: Bootstrap starting.
I20260805 22:50:34.739842 25643 tablet_bootstrap.cc:492] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: No bootstrap required, opened a new log
I20260805 22:50:34.740160 25643 ts_tablet_manager.cc:1397] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260805 22:50:34.742009 25647 tablet_bootstrap.cc:654] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:34.743048 25647 log.cc:824] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:34.745826 25647 tablet_bootstrap.cc:492] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: No bootstrap required, opened a new log
I20260805 22:50:34.746125 25647 ts_tablet_manager.cc:1397] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260805 22:50:34.749171 25640 raft_consensus.cc:348] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:50:34.749764 25640 raft_consensus.cc:374] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:34.749866 25640 raft_consensus.cc:729] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6cfd11a6ee4807b67f4f103e9631fd, State: Initialized, Role: FOLLOWER
I20260805 22:50:34.750670 25640 consensus_queue.cc:260] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:50:34.750792 25643 raft_consensus.cc:348] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:50:34.751266 25643 raft_consensus.cc:374] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:34.751385 25643 raft_consensus.cc:729] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb030ffcc794c31ac1176184e65b8cc, State: Initialized, Role: FOLLOWER
I20260805 22:50:34.752170 25643 consensus_queue.cc:260] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:50:34.754348 25643 ts_tablet_manager.cc:1428] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
I20260805 22:50:34.755455 25619 heartbeater.cc:503] Master 127.4.28.62:45197 was elected leader, sending a full tablet report...
I20260805 22:50:34.755962 25640 ts_tablet_manager.cc:1428] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260805 22:50:34.759197 25647 raft_consensus.cc:348] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:50:34.759722 25647 raft_consensus.cc:374] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:34.759826 25647 raft_consensus.cc:729] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92359575832c4c0bb4340ab51f4ec106, State: Initialized, Role: FOLLOWER
I20260805 22:50:34.760543 25647 consensus_queue.cc:260] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:50:34.761865 25413 heartbeater.cc:503] Master 127.4.28.62:45197 was elected leader, sending a full tablet report...
I20260805 22:50:34.762593 25647 ts_tablet_manager.cc:1428] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260805 22:50:34.774137 25653 raft_consensus.cc:482] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:50:34.774472 25653 raft_consensus.cc:504] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:50:34.776417 25653 leader_election.cc:288] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d6cfd11a6ee4807b67f4f103e9631fd (127.4.28.3:44223), 1bb030ffcc794c31ac1176184e65b8cc (127.4.28.1:33755)
I20260805 22:50:34.784036 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.776922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:48985 (local address 127.4.28.3:44223)
0805 22:50:34.777151 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.777158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.778108 (+ 950us) server_negotiation.cc:407] Connection header received
0805 22:50:34.778361 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.778368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.778447 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.778527 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:34.779175 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.780047 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.782289 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.783076 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.783207 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.783315 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.783659 (+ 344us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.783791 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 22:50:34.784041 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.777070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:35611 (local address 127.4.28.1:33755)
0805 22:50:34.777279 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.777286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.778242 (+ 956us) server_negotiation.cc:407] Connection header received
0805 22:50:34.778457 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.778461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.778535 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.778626 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:34.779338 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.780073 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.782213 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.783044 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.783169 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.783290 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.783648 (+ 358us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.783787 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:50:34.784202 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.776799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:44223 (local address 127.4.28.2:48985)
0805 22:50:34.777971 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:34.777992 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:34.778149 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:34.778567 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:34.778572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:34.778598 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:34.779008 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.779018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.780190 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.780196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.782116 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.782132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.783339 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.783345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.783491 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.783521 (+ 30us) client_negotiation.cc:769] Sending connection context
0805 22:50:34.783612 (+ 91us) client_negotiation.cc:240] Negotiation successful
0805 22:50:34.783745 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":64,"thread_start_us":722,"threads_started":1}
I20260805 22:50:34.784621 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.776799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33755 (local address 127.4.28.2:35611)
0805 22:50:34.778136 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:34.778153 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:34.778290 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:34.778672 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:34.778675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:34.778711 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:34.779174 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.779186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.780216 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.780220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.782028 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:34.782044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.783319 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.783328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:34.783475 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.783505 (+ 30us) client_negotiation.cc:769] Sending connection context
0805 22:50:34.783590 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 22:50:34.783730 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"mutex_wait_us":298,"spinlock_wait_cycles":23808,"thread_start_us":693,"threads_started":1}
I20260805 22:50:34.785250 25550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c912d82055460aa8defcebfd0d3d17" candidate_uuid: "92359575832c4c0bb4340ab51f4ec106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" is_pre_election: true
I20260805 22:50:34.785547 25141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c912d82055460aa8defcebfd0d3d17" candidate_uuid: "92359575832c4c0bb4340ab51f4ec106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb030ffcc794c31ac1176184e65b8cc" is_pre_election: true
I20260805 22:50:34.785847 25550 raft_consensus.cc:2454] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92359575832c4c0bb4340ab51f4ec106 in term 0.
I20260805 22:50:34.786160 25141 raft_consensus.cc:2454] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92359575832c4c0bb4340ab51f4ec106 in term 0.
I20260805 22:50:34.786625 25237 leader_election.cc:302] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6cfd11a6ee4807b67f4f103e9631fd, 92359575832c4c0bb4340ab51f4ec106; no voters:
I20260805 22:50:34.789723 25653 raft_consensus.cc:2781] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:50:34.789932 25653 raft_consensus.cc:482] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:50:34.790032 25653 raft_consensus.cc:3037] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:34.793284 25653 raft_consensus.cc:504] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:50:34.794004 25653 leader_election.cc:288] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [CANDIDATE]: Term 1 election: Requested vote from peers 8d6cfd11a6ee4807b67f4f103e9631fd (127.4.28.3:44223), 1bb030ffcc794c31ac1176184e65b8cc (127.4.28.1:33755)
I20260805 22:50:34.794677 25141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c912d82055460aa8defcebfd0d3d17" candidate_uuid: "92359575832c4c0bb4340ab51f4ec106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb030ffcc794c31ac1176184e65b8cc"
I20260805 22:50:34.794684 25550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c912d82055460aa8defcebfd0d3d17" candidate_uuid: "92359575832c4c0bb4340ab51f4ec106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd"
I20260805 22:50:34.794878 25141 raft_consensus.cc:3037] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:34.794914 25550 raft_consensus.cc:3037] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:34.797597 25141 raft_consensus.cc:2454] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92359575832c4c0bb4340ab51f4ec106 in term 1.
I20260805 22:50:34.797796 25550 raft_consensus.cc:2454] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92359575832c4c0bb4340ab51f4ec106 in term 1.
I20260805 22:50:34.798075 25239 leader_election.cc:302] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bb030ffcc794c31ac1176184e65b8cc, 92359575832c4c0bb4340ab51f4ec106; no voters:
I20260805 22:50:34.798434 25653 raft_consensus.cc:2781] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:50:34.798933 25653 raft_consensus.cc:686] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 1 LEADER]: Becoming Leader. State: Replica: 92359575832c4c0bb4340ab51f4ec106, State: Running, Role: LEADER
I20260805 22:50:34.799444 25653 consensus_queue.cc:237] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } }
I20260805 22:50:34.804669 24928 catalog_manager.cc:5495] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92359575832c4c0bb4340ab51f4ec106 (127.4.28.2). New cstate: current_term: 1 leader_uuid: "92359575832c4c0bb4340ab51f4ec106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92359575832c4c0bb4340ab51f4ec106" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36177 } health_report { overall_health: HEALTHY } } }
I20260805 22:50:34.835553 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.830071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.4.28.2:36177)
0805 22:50:34.830267 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.830272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.830291 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:50:34.830455 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.830459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.830560 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.830697 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:50:34.831048 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.831791 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.833164 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.833649 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.833750 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.834282 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:50:34.834812 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:50:34.834938 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.835114 (+ 176us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.835203 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 22:50:34.844926 25141 raft_consensus.cc:1261] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [term 1 FOLLOWER]: Refusing update from remote peer 92359575832c4c0bb4340ab51f4ec106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:34.844925 25550 raft_consensus.cc:1261] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 1 FOLLOWER]: Refusing update from remote peer 92359575832c4c0bb4340ab51f4ec106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:34.846053 25660 consensus_queue.cc:1035] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6cfd11a6ee4807b67f4f103e9631fd" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:34.846437 25653 consensus_queue.cc:1035] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bb030ffcc794c31ac1176184e65b8cc" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:34.857646 25664 mvcc.cc:205] Tried to move back new op lower bound from 7315334081914175488 to 7315334081742991360. Current Snapshot: MvccSnapshot[applied={T|T < 7315334081914175488}]
I20260805 22:50:34.876642 24925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:50:34.877053 24925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:50:34.881349 24925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Sending DeleteTablet for 3 replicas of tablet 37c912d82055460aa8defcebfd0d3d17
I20260805 22:50:34.882863 25306 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c912d82055460aa8defcebfd0d3d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:34 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:42004
I20260805 22:50:34.882851 25110 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c912d82055460aa8defcebfd0d3d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:34 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:46012
I20260805 22:50:34.883087 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 24980
I20260805 22:50:34.884447 25679 tablet_replica.cc:330] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: stopping tablet replica
I20260805 22:50:34.884600 25515 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c912d82055460aa8defcebfd0d3d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:34 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:58182
I20260805 22:50:34.884964 25679 raft_consensus.cc:2229] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:50:34.885159 25678 tablet_replica.cc:330] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: stopping tablet replica
I20260805 22:50:34.885739 25679 raft_consensus.cc:2258] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:34.886528 25678 raft_consensus.cc:2229] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:50:34.887126 25678 raft_consensus.cc:2258] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:34.889786 25678 ts_tablet_manager.cc:1905] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:34.893930 25679 ts_tablet_manager.cc:1905] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:34.897584 25678 ts_tablet_manager.cc:1918] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:34.897909 25678 log.cc:1196] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/37c912d82055460aa8defcebfd0d3d17
I20260805 22:50:34.898630 25678 ts_tablet_manager.cc:1939] T 37c912d82055460aa8defcebfd0d3d17 P 1bb030ffcc794c31ac1176184e65b8cc: Deleting consensus metadata
I20260805 22:50:34.901067 24910 catalog_manager.cc:4841] TS 1bb030ffcc794c31ac1176184e65b8cc (127.4.28.1:33755): tablet 37c912d82055460aa8defcebfd0d3d17 (table test-table [id=010e2855f7094745a59346041edcd70c]) successfully deleted
I20260805 22:50:34.904285 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:34.884752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.4.28.1:33755)
0805 22:50:34.884959 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:34.884966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:34.885115 (+ 149us) server_negotiation.cc:407] Connection header received
0805 22:50:34.885340 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:34.885345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:34.885433 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:34.885587 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:34.887772 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.888934 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.890891 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:34.891552 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:34.891694 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:34.895493 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:34.895512 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:34.895533 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:34.895589 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:34.899653 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:34.900694 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:34.900704 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:34.900707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:34.900887 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:34.901403 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:34.901408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:34.901411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:34.901963 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:34.902279 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:34.903637 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0805 22:50:34.903908 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:50:34.905017 25196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:34.905753 25685 tablet_replica.cc:330] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: stopping tablet replica
I20260805 22:50:34.906544 25679 ts_tablet_manager.cc:1918] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:34.907244 25685 raft_consensus.cc:2229] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:50:34.907553 25679 log.cc:1196] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/37c912d82055460aa8defcebfd0d3d17
I20260805 22:50:34.908329 25679 ts_tablet_manager.cc:1939] T 37c912d82055460aa8defcebfd0d3d17 P 92359575832c4c0bb4340ab51f4ec106: Deleting consensus metadata
I20260805 22:50:34.908906 25685 raft_consensus.cc:2258] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:34.911329 24908 catalog_manager.cc:4841] TS 92359575832c4c0bb4340ab51f4ec106 (127.4.28.2:36177): tablet 37c912d82055460aa8defcebfd0d3d17 (table test-table [id=010e2855f7094745a59346041edcd70c]) successfully deleted
I20260805 22:50:34.911722 25685 ts_tablet_manager.cc:1905] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:34.916743 25685 ts_tablet_manager.cc:1918] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:34.916934 25685 log.cc:1196] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/37c912d82055460aa8defcebfd0d3d17
I20260805 22:50:34.917423 25685 ts_tablet_manager.cc:1939] T 37c912d82055460aa8defcebfd0d3d17 P 8d6cfd11a6ee4807b67f4f103e9631fd: Deleting consensus metadata
I20260805 22:50:34.919111 24908 catalog_manager.cc:4841] TS 8d6cfd11a6ee4807b67f4f103e9631fd (127.4.28.3:44223): tablet 37c912d82055460aa8defcebfd0d3d17 (table test-table [id=010e2855f7094745a59346041edcd70c]) successfully deleted
I20260805 22:50:35.026073 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 24980
I20260805 22:50:35.049399 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 25209
I20260805 22:50:35.064852 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:35.050250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.4.28.2:36177)
0805 22:50:35.050453 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:35.050459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:35.050556 (+ 97us) server_negotiation.cc:407] Connection header received
0805 22:50:35.050767 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:35.050777 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:35.050864 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:35.050957 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:35.052405 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:35.053292 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:35.054791 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:35.055231 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:35.055345 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:35.058356 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:35.058379 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:35.058398 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:35.058443 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:35.061777 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:35.062568 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:35.062580 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:35.062583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:35.062738 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:35.063135 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:35.063140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:35.063142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:35.063553 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:35.063904 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:35.064246 (+ 342us) server_negotiation.cc:299] Negotiation successful
0805 22:50:35.064485 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260805 22:50:35.065663 25403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:35.182130 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 25209
I20260805 22:50:35.209113 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 25416
I20260805 22:50:35.225111 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:35.210063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.4.28.3:44223)
0805 22:50:35.210250 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:35.210257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:35.210280 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:50:35.210499 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:35.210505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:35.210589 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:35.210705 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:35.212251 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:35.213017 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:35.214574 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:35.215014 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:35.215146 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:35.218103 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:35.218124 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:35.218143 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:35.218192 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:35.221712 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:35.222753 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:35.222764 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:35.222766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:35.222933 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:35.223389 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:35.223395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:35.223397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:35.223823 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:35.224160 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:35.224496 (+ 336us) server_negotiation.cc:299] Negotiation successful
0805 22:50:35.224748 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 22:50:35.225837 25610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:35.316692 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 25416
I20260805 22:50:35.341243 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 24882
I20260805 22:50:35.342644 24972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:35.460750 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 24882
I20260805 22:50:35.492110 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45197
--webserver_interface=127.4.28.62
--webserver_port=36299
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:35.869309 25693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:35.869659 25693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:35.869762 25693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:35.877313 25693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:35.877447 25693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:35.877496 25693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:35.877586 25693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:35.877641 25693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:35.888068 25693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:35.889478 25693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:35.890901 25693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:35.903400 25699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:35.903421 25706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:35.903373 25700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:35.904263 25693 server_base.cc:1034] running on GCE node
I20260805 22:50:35.905118 25693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:35.906792 25693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:35.908018 25693 hybrid_clock.cc:648] HybridClock initialized: now 1785970235907952 us; error 55 us; skew 500 ppm
I20260805 22:50:35.908423 25693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:35 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970235, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:35.911787 25693 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:50:35.913565 25693 webserver.cc:460] Webserver started at http://127.4.28.62:36299/ using document root <none> and password file <none>
I20260805 22:50:35.914302 25693 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:35.914424 25693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:35.920568 25693 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260805 22:50:35.923900 25723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:35.925452 25693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:50:35.925689 25693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "209edec2d6ff49a994e44042d0c1ac62"
format_stamp: "Formatted at 2026-08-05 22:50:29 on dist-test-slave-7lbf"
server_key: "3f53d53afa123cff754f448a0c62c3c5"
server_key_iv: "bd2de377b4a42fa76dfc795560567885"
server_key_version: "encryptionkey@0"
I20260805 22:50:35.926748 25693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:35.959687 25693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:35.964108 25693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:35.964406 25693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:35.983294 25693 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:45197
I20260805 22:50:35.983314 25805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:45197 every 8 connection(s)
I20260805 22:50:35.984944 25693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:50:35.987689 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 25693
I20260805 22:50:35.988729 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:33755
--local_ip_for_outbound_sockets=127.4.28.1
--tserver_master_addrs=127.4.28.62:45197
--webserver_port=40103
--webserver_interface=127.4.28.1
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:50:35.994567 25807 sys_catalog.cc:263] Verifying existing consensus state
I20260805 22:50:35.999647 25807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Bootstrap starting.
I20260805 22:50:36.022765 25807 log.cc:824] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:36.041374 25807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:50:36.042007 25807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62: Bootstrap complete.
I20260805 22:50:36.051518 25807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:36.052495 25807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209edec2d6ff49a994e44042d0c1ac62, State: Initialized, Role: FOLLOWER
I20260805 22:50:36.053148 25807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:36.053395 25807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:50:36.053543 25807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:50:36.053725 25807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:50:36.058063 25807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:36.058615 25807 leader_election.cc:302] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 209edec2d6ff49a994e44042d0c1ac62; no voters:
I20260805 22:50:36.059288 25807 leader_election.cc:288] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260805 22:50:36.059453 25814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:50:36.060527 25814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [term 3 LEADER]: Becoming Leader. State: Replica: 209edec2d6ff49a994e44042d0c1ac62, State: Running, Role: LEADER
I20260805 22:50:36.061116 25814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } }
I20260805 22:50:36.061645 25807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:50:36.064010 25818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 209edec2d6ff49a994e44042d0c1ac62. Latest consensus state: current_term: 3 leader_uuid: "209edec2d6ff49a994e44042d0c1ac62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } } }
I20260805 22:50:36.064528 25818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:36.065065 25815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "209edec2d6ff49a994e44042d0c1ac62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "209edec2d6ff49a994e44042d0c1ac62" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45197 } } }
I20260805 22:50:36.065341 25815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 209edec2d6ff49a994e44042d0c1ac62 [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:36.067585 25823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:50:36.077962 25823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:50:36.079733 25823 catalog_manager.cc:1259] Loaded cluster ID: e04a850d9e02480892fb6ee2b4076e13
I20260805 22:50:36.079882 25823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260805 22:50:36.082419 25823 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa12c2479ad google::LogMessage::Fail() at ??:0
@ 0x7fa12c24a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa12c2474de google::LogMessage::Flush() at ??:0
@ 0x7fa12c248a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa141c7a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa141c77976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa141cff610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa141cff439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa12ee93345 kudu::ThreadPool::DispatchThread() at ??:0
I20260805 22:50:36.142684 25850 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fa12ee9691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa12ee96799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa12ee71eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa13214b6db start_thread at ??:0
@ 0x7fa12b0e161f clone at ??:0
W20260805 22:50:36.412572 25812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:36.412897 25812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:36.413028 25812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:36.420553 25812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:36.420691 25812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:36.420858 25812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:50:36.431766 25812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:33755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=40103
--tserver_master_addrs=127.4.28.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:36.433308 25812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:36.434924 25812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:36.449504 25875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:36.449469 25878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:36.449738 25874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:36.450752 25812 server_base.cc:1034] running on GCE node
I20260805 22:50:36.451525 25812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:36.452754 25812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:36.454017 25812 hybrid_clock.cc:648] HybridClock initialized: now 1785970236453952 us; error 48 us; skew 500 ppm
I20260805 22:50:36.454396 25812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:36 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970236, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:36.457921 25812 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:50:36.459553 25812 webserver.cc:460] Webserver started at http://127.4.28.1:40103/ using document root <none> and password file <none>
I20260805 22:50:36.460345 25812 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:36.460481 25812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:36.466550 25812 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:50:36.469663 25886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:36.471282 25812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 22:50:36.471462 25812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "1bb030ffcc794c31ac1176184e65b8cc"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "53567f229e58666cbfb01f1e52d02c2a"
server_key_iv: "6c4e622490ffcfc03e019a7d39a93432"
server_key_version: "encryptionkey@0"
I20260805 22:50:36.472553 25812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:36.494125 25812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:36.498808 25812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:36.499123 25812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:36.500242 25812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:36.501844 25812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:36.501963 25812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:36.502097 25812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:36.502179 25812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:36.536026 25812 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:33755
I20260805 22:50:36.536113 26053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:33755 every 8 connection(s)
I20260805 22:50:36.537606 25812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:50:36.542965 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:36.540834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:40849)
0805 22:50:36.541727 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:36.541974 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":342,"threads_started":1}
W20260805 22:50:36.543687 26054 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:45197 (0 consecutive failures): Network error: Failed to ping master at 127.4.28.62:45197: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
I20260805 22:50:36.544445 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 25812
I20260805 22:50:36.544668 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:36.544262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:35359)
0805 22:50:36.544436 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:36.544508 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260805 22:50:36.545241 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:36177
--local_ip_for_outbound_sockets=127.4.28.2
--tserver_master_addrs=127.4.28.62:45197
--webserver_port=46633
--webserver_interface=127.4.28.2
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:50:36.546042 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:36.545650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:46411)
0805 22:50:36.545799 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:36.545920 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260805 22:50:36.546274 26054 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 22:50:36.915333 26058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:36.915666 26058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:36.915812 26058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:36.923375 26058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:36.923525 26058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:36.923696 26058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:50:36.933993 26058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:36177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=46633
--tserver_master_addrs=127.4.28.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:36.935412 26058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:36.936828 26058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:36.950630 26066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:36.950531 26073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:36.950609 26068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:36.951260 26058 server_base.cc:1034] running on GCE node
I20260805 22:50:36.952061 26058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:36.953066 26058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:36.954270 26058 hybrid_clock.cc:648] HybridClock initialized: now 1785970236954221 us; error 35 us; skew 500 ppm
I20260805 22:50:36.954622 26058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:36 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970236, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:36.957954 26058 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:50:36.959517 26058 webserver.cc:460] Webserver started at http://127.4.28.2:46633/ using document root <none> and password file <none>
I20260805 22:50:36.960268 26058 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:36.960395 26058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:36.966645 26058 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:50:36.969864 26083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:36.971448 26058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:50:36.971661 26058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "92359575832c4c0bb4340ab51f4ec106"
format_stamp: "Formatted at 2026-08-05 22:50:30 on dist-test-slave-7lbf"
server_key: "9589e19b3d226bd07530ea78c918bbd8"
server_key_iv: "4825ba912ebb390a632bfd3b78858953"
server_key_version: "encryptionkey@0"
I20260805 22:50:36.972762 26058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:36.988219 26058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:36.992827 26058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:36.993127 26058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:36.994307 26058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:36.995779 26058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:36.995875 26058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:36.995963 26058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:36.996018 26058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:37.029001 26058 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:36177
I20260805 22:50:37.029095 26267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:36177 every 8 connection(s)
I20260805 22:50:37.030680 26058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:50:37.036087 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:37.033975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:32997)
0805 22:50:37.034821 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:37.035087 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":327,"threads_started":1}
W20260805 22:50:37.036825 26268 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:45197 (0 consecutive failures): Network error: Failed to ping master at 127.4.28.62:45197: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
I20260805 22:50:37.037664 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 26058
I20260805 22:50:37.037912 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:37.037420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:53951)
0805 22:50:37.037634 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:37.037733 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260805 22:50:37.038446 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:44223
--local_ip_for_outbound_sockets=127.4.28.3
--tserver_master_addrs=127.4.28.62:45197
--webserver_port=46729
--webserver_interface=127.4.28.3
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:50:37.039002 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:37.038539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:55375)
0805 22:50:37.038778 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:37.038867 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260805 22:50:37.039268 26268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 22:50:37.402119 26271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:37.402434 26271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:37.402534 26271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:37.410162 26271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:37.410296 26271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:37.410426 26271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:50:37.420744 26271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:44223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=46729
--tserver_master_addrs=127.4.28.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:37.422142 26271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:37.423542 26271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:37.436997 26283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:37.437059 26281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:37.437052 26280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:37.437788 26271 server_base.cc:1034] running on GCE node
I20260805 22:50:37.438445 26271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:37.439492 26271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:37.440730 26271 hybrid_clock.cc:648] HybridClock initialized: now 1785970237440655 us; error 60 us; skew 500 ppm
I20260805 22:50:37.441075 26271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:37 dist-test-slave-7lbf krb5kdc[24084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970237, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:37.444341 26271 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:50:37.445988 26271 webserver.cc:460] Webserver started at http://127.4.28.3:46729/ using document root <none> and password file <none>
I20260805 22:50:37.446770 26271 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:37.446893 26271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:37.452560 26271 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:50:37.455539 26293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:37.457027 26271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:50:37.457209 26271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "8d6cfd11a6ee4807b67f4f103e9631fd"
format_stamp: "Formatted at 2026-08-05 22:50:31 on dist-test-slave-7lbf"
server_key: "e8c7cdc7c43937669b6933d4908e08cc"
server_key_iv: "e035a3227781f03940189323554fe7c2"
server_key_version: "encryptionkey@0"
I20260805 22:50:37.458374 26271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:37.474409 26271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:37.479249 26271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:37.479530 26271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:37.480616 26271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:37.482189 26271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:37.482292 26271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:37.482401 26271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:37.482471 26271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:37.514734 26271 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:44223
I20260805 22:50:37.514797 26472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:44223 every 8 connection(s)
I20260805 22:50:37.516283 26271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:50:37.521414 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 26271
I20260805 22:50:37.521832 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:37.519489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:35739)
0805 22:50:37.520291 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:37.520525 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":299,"threads_started":1}
W20260805 22:50:37.522534 26473 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:45197 (0 consecutive failures): Network error: Failed to ping master at 127.4.28.62:45197: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
I20260805 22:50:37.523494 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:37.523088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:46991)
0805 22:50:37.523263 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:37.523346 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260805 22:50:37.524348 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:37.524040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:41369)
0805 22:50:37.524164 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:37.524241 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260805 22:50:37.524595 26473 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260805 22:50:37.547745 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:37.546922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:38239)
0805 22:50:37.547403 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:37.547495 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
I20260805 22:50:38.040727 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:38.039828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:46647)
0805 22:50:38.040385 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:38.040478 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":302,"threads_started":1}
I20260805 22:50:38.526341 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:38.525429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:53613)
0805 22:50:38.525989 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:38.526091 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":272,"threads_started":1}
I20260805 22:50:38.549692 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:38.548798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:35531)
0805 22:50:38.549369 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:38.549461 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":307,"threads_started":1}
I20260805 22:50:39.042665 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:39.041806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:40387)
0805 22:50:39.042363 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:39.042470 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":250,"threads_started":1}
I20260805 22:50:39.528326 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:39.527436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:36825)
0805 22:50:39.528047 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:39.528145 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":318,"threads_started":1}
I20260805 22:50:39.551375 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:39.550635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:34495)
0805 22:50:39.551111 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:39.551194 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260805 22:50:40.044451 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:40.043601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:53437)
0805 22:50:40.044168 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:40.044257 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":313,"threads_started":1}
I20260805 22:50:40.530385 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:40.529408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:38457)
0805 22:50:40.530085 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:40.530196 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":370,"threads_started":1}
I20260805 22:50:40.553252 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:40.552432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:46593)
0805 22:50:40.552974 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:40.553068 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":277,"threads_started":1}
I20260805 22:50:41.046429 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:41.045596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:53797)
0805 22:50:41.046102 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:41.046196 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":231,"threads_started":1}
I20260805 22:50:41.532249 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:41.531387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:59893)
0805 22:50:41.531944 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:41.532042 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":249,"threads_started":1}
I20260805 22:50:41.555207 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:41.554346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:60679)
0805 22:50:41.554917 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:41.555017 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":289,"threads_started":1}
I20260805 22:50:42.048391 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:42.047513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:41579)
0805 22:50:42.048106 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:42.048201 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":307,"threads_started":1}
I20260805 22:50:42.534307 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:42.533422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:50033)
0805 22:50:42.533982 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:42.534086 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":296,"threads_started":1}
I20260805 22:50:42.557235 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:42.556281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:59209)
0805 22:50:42.556845 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:42.556996 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":298,"threads_started":1}
I20260805 22:50:43.050647 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:43.049622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:59063)
0805 22:50:43.050298 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:43.050418 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":378,"threads_started":1}
I20260805 22:50:43.536422 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:43.535481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:49175)
0805 22:50:43.536038 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:43.536170 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":284,"threads_started":1}
I20260805 22:50:43.559209 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:43.558324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:52313)
0805 22:50:43.558904 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:43.558992 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":258,"threads_started":1}
I20260805 22:50:44.052655 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:44.051742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:44963)
0805 22:50:44.052329 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:44.052418 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":257,"threads_started":1}
I20260805 22:50:44.538635 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:44.537670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:35905)
0805 22:50:44.538290 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:44.538394 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":259,"threads_started":1}
I20260805 22:50:44.561043 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:44.560262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:48133)
0805 22:50:44.560743 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:44.560834 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":248,"threads_started":1}
I20260805 22:50:45.054575 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:45.053775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:44905)
0805 22:50:45.054292 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:45.054397 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":244,"threads_started":1}
I20260805 22:50:45.540603 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:45.539758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:48923)
0805 22:50:45.540321 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:45.540424 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":278,"threads_started":1}
I20260805 22:50:45.563172 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:45.562204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:54061)
0805 22:50:45.562836 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:45.562944 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":343,"threads_started":1}
I20260805 22:50:46.056447 26515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:46.055582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:44381)
0805 22:50:46.056158 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:46.056246 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":293,"threads_started":1}
I20260805 22:50:46.542649 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:46.541727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:51669)
0805 22:50:46.542307 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:46.542403 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":323,"threads_started":1}
I20260805 22:50:46.565248 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:46.564333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:55317)
0805 22:50:46.564909 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:46.565030 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":274,"threads_started":1}
I20260805 22:50:47.058535 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:47.057603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:36421)
0805 22:50:47.058180 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:47.058302 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":271,"threads_started":1}
I20260805 22:50:47.544567 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:47.543724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:37573)
0805 22:50:47.544295 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:47.544390 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":334,"threads_started":1}
I20260805 22:50:47.567121 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:47.566335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:49689)
0805 22:50:47.566804 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:47.566907 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260805 22:50:48.060643 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:48.059728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:41597)
0805 22:50:48.060310 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:48.060439 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":301,"threads_started":1}
I20260805 22:50:48.546514 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:48.545661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:37467)
0805 22:50:48.546212 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:48.546321 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":264,"threads_started":1}
I20260805 22:50:48.569056 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:48.568217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:53631)
0805 22:50:48.568770 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:48.568878 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":278,"threads_started":1}
I20260805 22:50:49.062801 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:49.061891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:54433)
0805 22:50:49.062478 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:49.062589 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":265,"threads_started":1}
I20260805 22:50:49.548300 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:49.547542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:60873)
0805 22:50:49.548033 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:49.548127 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":220,"threads_started":1}
I20260805 22:50:49.571013 26530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:49.570092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:52967)
0805 22:50:49.570704 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:49.570812 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":316,"threads_started":1}
I20260805 22:50:50.064755 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:50.063862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:54147)
0805 22:50:50.064424 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:50.064532 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":270,"threads_started":1}
I20260805 22:50:50.550163 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:50.549296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:60617)
0805 22:50:50.549851 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:50.549946 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":281,"threads_started":1}
I20260805 22:50:50.572944 26537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:50.572109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:57601)
0805 22:50:50.572600 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:50.572691 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":244,"threads_started":1}
I20260805 22:50:51.066543 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:51.065735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:42225)
0805 22:50:51.066229 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:51.066335 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":225,"threads_started":1}
I20260805 22:50:51.552062 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:51.551187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:50713)
0805 22:50:51.551717 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:51.551817 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":236,"threads_started":1}
I20260805 22:50:51.575070 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:51.574269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.1:42361)
0805 22:50:51.574760 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:51.574862 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":246,"threads_started":1}
I20260805 22:50:52.068689 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:52.067806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.2:60221)
0805 22:50:52.068353 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:52.068478 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":274,"threads_started":1}
I20260805 22:50:52.522785 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 25812
I20260805 22:50:52.547026 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:52.524139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.4.28.1:33755)
0805 22:50:52.524843 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:52.524860 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:52.524902 (+ 42us) server_negotiation.cc:407] Connection header received
0805 22:50:52.526504 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:52.526523 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:52.526974 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:52.527327 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:52.528879 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:52.530223 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:52.531575 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:52.532300 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:52.532499 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:52.539752 (+ 7253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:52.539764 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:52.539785 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:52.539834 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:52.543137 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:52.544058 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:52.544069 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:52.544078 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:52.544283 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:52.544761 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:52.544765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:52.544767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:52.545385 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:52.545720 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:52.546234 (+ 514us) server_negotiation.cc:299] Negotiation successful
0805 22:50:52.546718 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":339,"threads_started":1}
I20260805 22:50:52.548573 26046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:52.554180 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:52.553385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45197 (local address 127.4.28.3:34899)
0805 22:50:52.553878 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:52.553982 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:45197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":251,"threads_started":1}
I20260805 22:50:52.634552 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 25812
I20260805 22:50:52.661981 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 26058
I20260805 22:50:52.681433 26549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:52.663111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.4.28.2:36177)
0805 22:50:52.663705 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:52.663721 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:52.663762 (+ 41us) server_negotiation.cc:407] Connection header received
0805 22:50:52.665323 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:52.665336 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:52.665885 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:52.666225 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:52.667725 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:52.669096 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:52.670691 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:52.671491 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:52.671654 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:52.674560 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:52.674570 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:52.674590 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:52.674642 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:52.677993 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:52.678817 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:52.678826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:52.678834 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:52.679057 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:52.679417 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:52.679420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:52.679422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:52.679999 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:52.680282 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:52.680662 (+ 380us) server_negotiation.cc:299] Negotiation successful
0805 22:50:52.681131 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":256,"threads_started":1}
I20260805 22:50:52.682951 26259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:52.768399 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 26058
I20260805 22:50:52.797489 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 26271
I20260805 22:50:52.816573 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:52.798593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.4.28.3:44223)
0805 22:50:52.799311 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:52.799326 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:52.799383 (+ 57us) server_negotiation.cc:407] Connection header received
0805 22:50:52.801038 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:52.801050 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:52.801577 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:52.801922 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:52.803410 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:52.804717 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:52.806118 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:52.806843 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:52.806994 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:52.809708 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:52.809718 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:52.809738 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:52.809786 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:52.812730 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:52.813658 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:52.813669 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:52.813681 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:52.813949 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:52.814427 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:52.814432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:52.814434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:52.815060 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:52.815325 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:52.815827 (+ 502us) server_negotiation.cc:299] Negotiation successful
0805 22:50:52.816244 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":352,"threads_started":1}
I20260805 22:50:52.818188 26466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:52.907280 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 26271
2026-08-05T22:50:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25262 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:50:53 dist-test-slave-7lbf krb5kdc[26557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:50:53 dist-test-slave-7lbf krb5kdc[26557](info): set up 2 sockets
Aug 05 22:50:53 dist-test-slave-7lbf krb5kdc[26557](info): commencing operation
krb5kdc: starting...
W20260805 22:50:54.141053 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:50:54 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:50:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:50:54.473670 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34627
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:34627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:54.844484 26588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:54.844838 26588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:54.844986 26588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:54.852576 26588 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260805 22:50:54.852702 26588 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260805 22:50:54.852741 26588 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260805 22:50:54.852814 26588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:50:54.852900 26588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:54.853001 26588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:54.853060 26588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:50:54.853138 26588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:50:54.863766 26588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:34627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:54.865265 26588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:54.866880 26588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:54.880352 26596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:54.880332 26599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:54.880244 26595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:54.881002 26588 server_base.cc:1034] running on GCE node
I20260805 22:50:54.881987 26588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:54.883665 26588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:54.884967 26588 hybrid_clock.cc:648] HybridClock initialized: now 1785970254884854 us; error 47 us; skew 500 ppm
I20260805 22:50:54.885372 26588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:54 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970254, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:54.888971 26588 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:50:54.890820 26588 webserver.cc:460] Webserver started at http://127.4.28.62:41965/ using document root <none> and password file <none>
I20260805 22:50:54.891662 26588 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:54.891782 26588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:54.892143 26588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:54.895610 26588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f"
format_stamp: "Formatted at 2026-08-05 22:50:54 on dist-test-slave-7lbf"
server_key: "bf8dab32a3208b2f9e3f1046e3deff4a"
server_key_iv: "b36c7465511365d2d119de166568bcba"
server_key_version: "encryptionkey@0"
I20260805 22:50:54.896565 26588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f"
format_stamp: "Formatted at 2026-08-05 22:50:54 on dist-test-slave-7lbf"
server_key: "bf8dab32a3208b2f9e3f1046e3deff4a"
server_key_iv: "b36c7465511365d2d119de166568bcba"
server_key_version: "encryptionkey@0"
I20260805 22:50:54.902681 26588 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260805 22:50:54.906840 26610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:54.908381 26588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:50:54.908583 26588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f"
format_stamp: "Formatted at 2026-08-05 22:50:54 on dist-test-slave-7lbf"
server_key: "bf8dab32a3208b2f9e3f1046e3deff4a"
server_key_iv: "b36c7465511365d2d119de166568bcba"
server_key_version: "encryptionkey@0"
I20260805 22:50:54.908818 26588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:54.951499 26588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:54.956197 26588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:54.956497 26588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:54.976097 26588 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:34627
I20260805 22:50:54.976140 26683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:34627 every 8 connection(s)
I20260805 22:50:54.977662 26588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:50:54.978068 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 26588
I20260805 22:50:54.978282 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:50:54.979107 4208 external_mini_cluster.cc:1428] Setting key 95a78118890aa105b4153a6cc9f4d560
I20260805 22:50:54.982710 26688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:50:54 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:54.996865 26688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Bootstrap starting.
I20260805 22:50:55.000931 26688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:55.002516 26688 log.cc:824] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:55.006536 26688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: No bootstrap required, opened a new log
I20260805 22:50:55.006646 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:54.981360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.4.28.62:34627)
0805 22:50:54.982265 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:54.982284 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:54.982338 (+ 54us) server_negotiation.cc:407] Connection header received
0805 22:50:54.984666 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:54.984699 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:54.985262 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:54.985764 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:54.987312 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:54.989508 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:54.991201 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:54.992274 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:54.992489 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:54.997654 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:54.997666 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:54.997686 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:54.997737 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:55.001213 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:55.002320 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:55.002328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:55.002337 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:55.002553 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:55.003151 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:55.003158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:55.003161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:55.004104 (+ 943us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:55.004394 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:55.005043 (+ 649us) server_negotiation.cc:299] Negotiation successful
0805 22:50:55.005499 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":346,"threads_started":1}
I20260805 22:50:55.015333 26688 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:50:55.015803 26688 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:55.015915 26688 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fdc5ff3fcd48378a29acfdc0f2710f, State: Initialized, Role: FOLLOWER
I20260805 22:50:55.016659 26688 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:50:55.016985 26688 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:50:55.017127 26688 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:50:55.017303 26688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:55.019737 26688 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:50:55.020351 26688 leader_election.cc:302] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1fdc5ff3fcd48378a29acfdc0f2710f; no voters:
I20260805 22:50:55.021236 26688 leader_election.cc:288] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:50:55.021378 26699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:50:55.022230 26699 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 1 LEADER]: Becoming Leader. State: Replica: a1fdc5ff3fcd48378a29acfdc0f2710f, State: Running, Role: LEADER
I20260805 22:50:55.022845 26699 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:50:55.023584 26688 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:50:55.025933 26702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fdc5ff3fcd48378a29acfdc0f2710f. Latest consensus state: current_term: 1 leader_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } } }
I20260805 22:50:55.026533 26702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:55.027037 26700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } } }
I20260805 22:50:55.027347 26700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: This master's current role is: LEADER
I20260805 22:50:55.028736 26711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:50:55.034682 26711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:50:55.044513 26711 catalog_manager.cc:1347] Generated new cluster ID: 729745043b6342dcba34024a453b8f3e
I20260805 22:50:55.044662 26711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:50:55.057387 26711 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:50:55.059453 26711 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:50:55.075047 26711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Generated new TSK 0
I20260805 22:50:55.075997 26711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:50:55.171458 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34627
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:55.545006 26727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:55.545325 26727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:55.545457 26727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:55.553020 26727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:55.553162 26727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:55.553326 26727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:50:55.563733 26727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:55.565179 26727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:55.566663 26727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:55.580600 26734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:55.580581 26735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:55.580895 26737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:55.581683 26727 server_base.cc:1034] running on GCE node
I20260805 22:50:55.582357 26727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:55.583390 26727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:55.584612 26727 hybrid_clock.cc:648] HybridClock initialized: now 1785970255584551 us; error 52 us; skew 500 ppm
I20260805 22:50:55.584959 26727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:55 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:55.588375 26727 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:50:55.590029 26727 webserver.cc:460] Webserver started at http://127.4.28.1:42595/ using document root <none> and password file <none>
I20260805 22:50:55.590790 26727 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:55.590919 26727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:55.591264 26727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:55.594599 26727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "21d417c90743455aa270caa2dfccba13"
format_stamp: "Formatted at 2026-08-05 22:50:55 on dist-test-slave-7lbf"
server_key: "8848b9239fe5bbf32360cbbef7697da6"
server_key_iv: "8c596d9d3e988ff94e652c4578fb5522"
server_key_version: "encryptionkey@0"
I20260805 22:50:55.595422 26727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "21d417c90743455aa270caa2dfccba13"
format_stamp: "Formatted at 2026-08-05 22:50:55 on dist-test-slave-7lbf"
server_key: "8848b9239fe5bbf32360cbbef7697da6"
server_key_iv: "8c596d9d3e988ff94e652c4578fb5522"
server_key_version: "encryptionkey@0"
I20260805 22:50:55.601282 26727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:50:55.605227 26747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:55.606834 26727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:50:55.607028 26727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "21d417c90743455aa270caa2dfccba13"
format_stamp: "Formatted at 2026-08-05 22:50:55 on dist-test-slave-7lbf"
server_key: "8848b9239fe5bbf32360cbbef7697da6"
server_key_iv: "8c596d9d3e988ff94e652c4578fb5522"
server_key_version: "encryptionkey@0"
I20260805 22:50:55.607239 26727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:55.640239 26727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:55.647023 26727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:55.647485 26727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:55.649163 26727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:55.651556 26727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:55.651696 26727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:55.651881 26727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:55.652001 26727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:55.701879 26727 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:35985
I20260805 22:50:55.701959 26917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:35985 every 8 connection(s)
I20260805 22:50:55.704074 26727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:50:55.711212 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 26727
I20260805 22:50:55.711639 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:50:55.712859 4208 external_mini_cluster.cc:1428] Setting key a2629309b5cf91d9094ae194dd43578c
Aug 05 22:50:55 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:55.738695 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:55.708252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:54995 (local address 127.4.28.62:34627)
0805 22:50:55.708946 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:55.708957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:55.710712 (+ 1755us) server_negotiation.cc:407] Connection header received
0805 22:50:55.714218 (+ 3506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:55.714225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:55.714372 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:55.714507 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:55.716860 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:55.717958 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:55.720359 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:55.721073 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:55.721316 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:55.729017 (+ 7701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:55.729030 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:55.729034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:55.729105 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:55.733143 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:55.734692 (+ 1549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:55.734701 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:55.734704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:55.734968 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:55.735664 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:55.735670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:55.735686 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:55.736315 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:55.736534 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:55.737700 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0805 22:50:55.738064 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":352,"threads_started":1}
I20260805 22:50:55.739697 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:55.709208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:54995)
0805 22:50:55.710448 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:55.710531 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:55.713739 (+ 3208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:55.714946 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:55.714960 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:55.715693 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:55.716573 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:55.716602 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:55.718188 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:55.718192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:55.720157 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:55.720178 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:55.721621 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:55.721638 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:55.722253 (+ 615us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:55.723575 (+ 1322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:55.723659 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:55.728553 (+ 4894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:55.733417 (+ 4864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:55.733438 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:55.733470 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:55.734472 (+ 1002us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:55.735133 (+ 661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:55.735147 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:55.735149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:55.735475 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:55.736510 (+ 1035us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:55.736524 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:55.737067 (+ 543us) client_negotiation.cc:769] Sending connection context
0805 22:50:55.737441 (+ 374us) client_negotiation.cc:240] Negotiation successful
0805 22:50:55.738136 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":517,"threads_started":1}
I20260805 22:50:55.741950 26918 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34627
I20260805 22:50:55.742511 26918 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:55.743891 26918 heartbeater.cc:511] Master 127.4.28.62:34627 requested a full tablet report, sending...
I20260805 22:50:55.747884 26637 ts_manager.cc:194] Registered new tserver with Master: 21d417c90743455aa270caa2dfccba13 (127.4.28.1:35985)
I20260805 22:50:55.750407 26637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:54995
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:50:55.814225 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34627
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:56.502700 26927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:56.503350 26927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:56.503636 26927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:56.517827 26927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:56.518122 26927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:56.518383 26927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:50:56.536674 26927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:56.539610 26927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:56.542145 26927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:56.564926 26934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:56.564818 26935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:56.564900 26940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:56.567106 26927 server_base.cc:1034] running on GCE node
I20260805 22:50:56.568190 26927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:56.570024 26927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:56.571447 26927 hybrid_clock.cc:648] HybridClock initialized: now 1785970256571338 us; error 89 us; skew 500 ppm
I20260805 22:50:56.572081 26927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:56 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970256, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:56.578069 26927 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:50:56.581040 26927 webserver.cc:460] Webserver started at http://127.4.28.2:41449/ using document root <none> and password file <none>
I20260805 22:50:56.582409 26927 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:56.582676 26927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:56.583266 26927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:56.588910 26927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "80fff5363cf9449dbfe7cc060b19165a"
format_stamp: "Formatted at 2026-08-05 22:50:56 on dist-test-slave-7lbf"
server_key: "670640efc9bb08f70efc62ca17e175b9"
server_key_iv: "c8880aa68abe617736e2384929f1c4da"
server_key_version: "encryptionkey@0"
I20260805 22:50:56.590332 26927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "80fff5363cf9449dbfe7cc060b19165a"
format_stamp: "Formatted at 2026-08-05 22:50:56 on dist-test-slave-7lbf"
server_key: "670640efc9bb08f70efc62ca17e175b9"
server_key_iv: "c8880aa68abe617736e2384929f1c4da"
server_key_version: "encryptionkey@0"
I20260805 22:50:56.600788 26927 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260805 22:50:56.607978 26951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:56.610637 26927 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260805 22:50:56.610929 26927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "80fff5363cf9449dbfe7cc060b19165a"
format_stamp: "Formatted at 2026-08-05 22:50:56 on dist-test-slave-7lbf"
server_key: "670640efc9bb08f70efc62ca17e175b9"
server_key_iv: "c8880aa68abe617736e2384929f1c4da"
server_key_version: "encryptionkey@0"
I20260805 22:50:56.611361 26927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:56.667114 26927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:56.674499 26927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:56.675019 26927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:56.676677 26927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:56.679132 26927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:56.679275 26927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:56.679508 26927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:56.679582 26927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:56.735493 26927 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:39975
I20260805 22:50:56.735631 27125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:39975 every 8 connection(s)
I20260805 22:50:56.738050 26927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:50:56.741694 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 26927
I20260805 22:50:56.742089 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:50:56.742944 4208 external_mini_cluster.cc:1428] Setting key 4d2c6ac5e39122dd24d648e03dcb5f93
I20260805 22:50:56.757493 26918 heartbeater.cc:503] Master 127.4.28.62:34627 was elected leader, sending a full tablet report...
Aug 05 22:50:56 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970256, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:56.780165 27129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:56.743183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:51023 (local address 127.4.28.62:34627)
0805 22:50:56.744105 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:56.744119 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:56.746681 (+ 2562us) server_negotiation.cc:407] Connection header received
0805 22:50:56.749995 (+ 3314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:56.750015 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:56.750194 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:56.750371 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:56.753345 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:56.754588 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:56.758603 (+ 4015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:56.759987 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:56.760237 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:56.769062 (+ 8825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:56.769071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:56.769084 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:56.769155 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:56.773940 (+ 4785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:56.775552 (+ 1612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:56.775558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:56.775561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:56.775803 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:56.776642 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:56.776655 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:56.776658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:56.777199 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:56.777474 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:56.778945 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0805 22:50:56.779498 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":426,"threads_started":1}
I20260805 22:50:56.782145 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:56.744610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:51023)
0805 22:50:56.746379 (+ 1769us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:56.746489 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:56.749409 (+ 2920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:56.750894 (+ 1485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:56.750906 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:56.751899 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:56.752965 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:56.753000 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:56.754880 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:56.754893 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:56.758198 (+ 3305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:56.758231 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:56.760942 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:56.760961 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:56.761553 (+ 592us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:56.762866 (+ 1313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:56.762939 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:56.768401 (+ 5462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:56.774338 (+ 5937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:56.774369 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:56.774388 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:56.775259 (+ 871us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:56.776031 (+ 772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:56.776036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:56.776038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:56.776427 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:56.777432 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:56.777455 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:56.778240 (+ 785us) client_negotiation.cc:769] Sending connection context
0805 22:50:56.778714 (+ 474us) client_negotiation.cc:240] Negotiation successful
0805 22:50:56.779899 (+ 1185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"thread_start_us":613,"threads_started":1}
I20260805 22:50:56.784940 27127 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34627
I20260805 22:50:56.785707 27127 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:56.787370 27127 heartbeater.cc:511] Master 127.4.28.62:34627 requested a full tablet report, sending...
I20260805 22:50:56.790764 26637 ts_manager.cc:194] Registered new tserver with Master: 80fff5363cf9449dbfe7cc060b19165a (127.4.28.2:39975)
I20260805 22:50:56.792675 26637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:51023
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
W20260805 22:50:56.852334 4208 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.28.3: real 0.104s user 0.001s sys 0.041s
I20260805 22:50:56.853626 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34627
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:50:57.480794 27136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:50:57.481397 27136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:50:57.481696 27136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:50:57.494705 27136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:50:57.495004 27136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:50:57.495461 27136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:50:57.513664 27136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:50:57.516250 27136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:50:57.518883 27136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:50:57.543988 27145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:57.544077 27146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:50:57.544835 27148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:50:57.546205 27136 server_base.cc:1034] running on GCE node
I20260805 22:50:57.547467 27136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:50:57.549424 27136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:50:57.551028 27136 hybrid_clock.cc:648] HybridClock initialized: now 1785970257550893 us; error 137 us; skew 500 ppm
I20260805 22:50:57.551668 27136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:50:57 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970257, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:50:57.561488 27136 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:50:57.564545 27136 webserver.cc:460] Webserver started at http://127.4.28.3:45157/ using document root <none> and password file <none>
I20260805 22:50:57.566006 27136 fs_manager.cc:359] Metadata directory not provided
I20260805 22:50:57.566246 27136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:50:57.566829 27136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:50:57.572578 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "c74d9fc31fc24d70ab696c37509470b1"
format_stamp: "Formatted at 2026-08-05 22:50:57 on dist-test-slave-7lbf"
server_key: "a6b5928e0b7acb5a250357ae9ac11be7"
server_key_iv: "4dbe66b65414a8a5858099a3fcf3b012"
server_key_version: "encryptionkey@0"
I20260805 22:50:57.574055 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "c74d9fc31fc24d70ab696c37509470b1"
format_stamp: "Formatted at 2026-08-05 22:50:57 on dist-test-slave-7lbf"
server_key: "a6b5928e0b7acb5a250357ae9ac11be7"
server_key_iv: "4dbe66b65414a8a5858099a3fcf3b012"
server_key_version: "encryptionkey@0"
I20260805 22:50:57.584937 27136 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260805 22:50:57.592332 27157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:57.595284 27136 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260805 22:50:57.595634 27136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "c74d9fc31fc24d70ab696c37509470b1"
format_stamp: "Formatted at 2026-08-05 22:50:57 on dist-test-slave-7lbf"
server_key: "a6b5928e0b7acb5a250357ae9ac11be7"
server_key_iv: "4dbe66b65414a8a5858099a3fcf3b012"
server_key_version: "encryptionkey@0"
I20260805 22:50:57.595969 27136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:50:57.622136 27136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:50:57.630895 27136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:50:57.631505 27136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:50:57.633591 27136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:50:57.636359 27136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:50:57.636488 27136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:57.636694 27136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:50:57.636781 27136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:50:57.696020 27136 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:37095
I20260805 22:50:57.696126 27344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:37095 every 8 connection(s)
I20260805 22:50:57.698511 27136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:50:57.701263 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 27136
I20260805 22:50:57.701694 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:50:57.702615 4208 external_mini_cluster.cc:1428] Setting key 8c9fb8a42150e1700f297d84b0eb31cd
Aug 05 22:50:57 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970257, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:50:57.745162 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:57.705033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:34367 (local address 127.4.28.62:34627)
0805 22:50:57.706212 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:57.706233 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:57.708424 (+ 2191us) server_negotiation.cc:407] Connection header received
0805 22:50:57.712851 (+ 4427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:57.712866 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:57.713136 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:57.713322 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:57.716865 (+ 3543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.718325 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.721805 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.722821 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.723123 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:57.730444 (+ 7321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:57.730460 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:57.730473 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:57.730613 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:57.737750 (+ 7137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:57.739799 (+ 2049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:57.739835 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:57.739853 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:57.740360 (+ 507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:57.741185 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:57.741191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:57.741195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:57.742075 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:57.742391 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:57.743878 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0805 22:50:57.744388 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":725,"threads_started":1}
I20260805 22:50:57.746619 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:57.706311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:34367)
0805 22:50:57.708092 (+ 1781us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:57.708192 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:57.712194 (+ 4002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:57.714053 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:57.714078 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:57.715195 (+ 1117us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:50:57.716455 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:57.716504 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.718722 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.718726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:57.721574 (+ 2848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:57.721605 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.723397 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.723419 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:57.724026 (+ 607us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:57.725268 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:50:57.725327 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:50:57.729996 (+ 4669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:50:57.738051 (+ 8055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:57.738067 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:57.738110 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:57.739525 (+ 1415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:57.740573 (+ 1048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:50:57.740576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:50:57.740579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:50:57.740984 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:50:57.742308 (+ 1324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:50:57.742325 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:50:57.742966 (+ 641us) client_negotiation.cc:769] Sending connection context
0805 22:50:57.743516 (+ 550us) client_negotiation.cc:240] Negotiation successful
0805 22:50:57.744287 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":671,"threads_started":1}
I20260805 22:50:57.749817 27345 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34627
I20260805 22:50:57.750442 27345 heartbeater.cc:464] Registering TS with master...
I20260805 22:50:57.752107 27345 heartbeater.cc:511] Master 127.4.28.62:34627 requested a full tablet report, sending...
I20260805 22:50:57.754910 26637 ts_manager.cc:194] Registered new tserver with Master: c74d9fc31fc24d70ab696c37509470b1 (127.4.28.3:37095)
I20260805 22:50:57.756573 26637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:34367
I20260805 22:50:57.767910 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:50:57.799574 27127 heartbeater.cc:503] Master 127.4.28.62:34627 was elected leader, sending a full tablet report...
I20260805 22:50:57.805802 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:57.777614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.4.28.62:34627)
0805 22:50:57.777903 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:57.777910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:57.778822 (+ 912us) server_negotiation.cc:407] Connection header received
0805 22:50:57.779100 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:57.779113 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:57.779265 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:57.779398 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:57.782070 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.783365 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.786499 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.787345 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.787603 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:57.792339 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:57.792350 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:57.792356 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:57.792483 (+ 127us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:57.798987 (+ 6504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:57.801070 (+ 2083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:57.801075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:57.801087 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:57.801357 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:57.802491 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:57.802508 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:57.802512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:57.803385 (+ 873us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:57.803898 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:57.804788 (+ 890us) server_negotiation.cc:299] Negotiation successful
0805 22:50:57.805249 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260805 22:50:57.814934 26637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:50:57.831471 26637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:50:57.894928 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:57.858147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.4.28.2:39975)
0805 22:50:57.867095 (+ 8948us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:57.867130 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:57.877771 (+ 10641us) server_negotiation.cc:407] Connection header received
0805 22:50:57.878232 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:57.878255 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:57.878713 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:57.879067 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:57.880743 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.883037 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.888420 (+ 5383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.890891 (+ 2471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.891175 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:57.891830 (+ 655us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:57.893633 (+ 1803us) server_negotiation.cc:299] Negotiation successful
0805 22:50:57.894023 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8547,"thread_start_us":8263,"threads_started":1}
I20260805 22:50:57.895860 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:57.856804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:37095 (local address 127.0.0.1:34346)
0805 22:50:57.875534 (+ 18730us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:57.875673 (+ 139us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:57.876183 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:57.879318 (+ 3135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:57.879324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:57.879430 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:57.880449 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:57.880489 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.884473 (+ 3984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.884480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:57.889285 (+ 4805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:57.889324 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.893645 (+ 4321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.893676 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:57.894037 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:57.894111 (+ 74us) client_negotiation.cc:769] Sending connection context
0805 22:50:57.894366 (+ 255us) client_negotiation.cc:240] Negotiation successful
0805 22:50:57.894914 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18099,"mutex_wait_us":79,"thread_start_us":20198,"threads_started":1}
I20260805 22:50:57.896591 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:57.857213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.4.28.3:37095)
0805 22:50:57.874758 (+ 17545us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:57.874784 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:57.875936 (+ 1152us) server_negotiation.cc:407] Connection header received
0805 22:50:57.876816 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:57.876829 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:57.877395 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:57.877865 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:57.880941 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.884089 (+ 3148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.889740 (+ 5651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.893132 (+ 3392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.893414 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:57.893939 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:57.895176 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0805 22:50:57.895573 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17175,"thread_start_us":16177,"threads_started":1}
I20260805 22:50:57.897662 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:57.858070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.0.0.1:36620)
0805 22:50:57.877547 (+ 19477us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:57.877601 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:57.877894 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:57.881476 (+ 3582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:57.881503 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:57.881635 (+ 132us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:57.882577 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:57.882608 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.886303 (+ 3695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.886323 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:57.890492 (+ 4169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:57.890562 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.894548 (+ 3986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.894558 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:57.895054 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:57.895089 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:50:57.895596 (+ 507us) client_negotiation.cc:240] Negotiation successful
0805 22:50:57.896414 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19219,"mutex_wait_us":364,"spinlock_wait_cycles":35584,"thread_start_us":20340,"threads_started":1}
I20260805 22:50:57.898089 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:57.858561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.4.28.1:35985)
0805 22:50:57.878956 (+ 20395us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:57.878965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:57.879008 (+ 43us) server_negotiation.cc:407] Connection header received
0805 22:50:57.879181 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:57.880063 (+ 882us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:57.880649 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:57.881072 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:57.883006 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.885915 (+ 2909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.891038 (+ 5123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.894114 (+ 3076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.894367 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:57.894934 (+ 567us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:57.896408 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0805 22:50:57.896746 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20058,"thread_start_us":21002,"threads_started":1}
I20260805 22:50:57.899055 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:57.857043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:39975 (local address 127.0.0.1:57252)
0805 22:50:57.877482 (+ 20439us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:57.877549 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:57.877885 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:57.879097 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:57.879116 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:57.879176 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:57.880267 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:57.880285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.883381 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.883388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:57.888039 (+ 4651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:57.888078 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:57.892263 (+ 4185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:57.892284 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:57.892902 (+ 618us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:57.893066 (+ 164us) client_negotiation.cc:769] Sending connection context
0805 22:50:57.895740 (+ 2674us) client_negotiation.cc:240] Negotiation successful
0805 22:50:57.898146 (+ 2406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20178,"mutex_wait_us":70,"spinlock_wait_cycles":2816,"thread_start_us":21085,"threads_started":1}
I20260805 22:50:57.906723 27025 tablet_service.cc:1467] Processing CreateTablet for tablet 48bc23509a0749189e3fe8c7156ff9fa (DEFAULT_TABLE table=test-table [id=4ded93f334ce45c6b1f470d42c9416e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:57.906774 26824 tablet_service.cc:1467] Processing CreateTablet for tablet 48bc23509a0749189e3fe8c7156ff9fa (DEFAULT_TABLE table=test-table [id=4ded93f334ce45c6b1f470d42c9416e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:57.906725 27249 tablet_service.cc:1467] Processing CreateTablet for tablet 48bc23509a0749189e3fe8c7156ff9fa (DEFAULT_TABLE table=test-table [id=4ded93f334ce45c6b1f470d42c9416e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:50:57.909909 27025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bc23509a0749189e3fe8c7156ff9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:57.910246 27249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bc23509a0749189e3fe8c7156ff9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:57.910228 26824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bc23509a0749189e3fe8c7156ff9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:50:57.936897 27373 tablet_bootstrap.cc:492] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: Bootstrap starting.
I20260805 22:50:57.943830 27375 tablet_bootstrap.cc:492] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: Bootstrap starting.
I20260805 22:50:57.946264 27373 tablet_bootstrap.cc:654] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:57.947288 27376 tablet_bootstrap.cc:492] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Bootstrap starting.
I20260805 22:50:57.949054 27373 log.cc:824] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:57.951563 27375 tablet_bootstrap.cc:654] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:57.953364 27375 log.cc:824] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:57.954097 27376 tablet_bootstrap.cc:654] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Neither blocks nor log segments found. Creating new log.
I20260805 22:50:57.955777 27373 tablet_bootstrap.cc:492] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: No bootstrap required, opened a new log
I20260805 22:50:57.956466 27373 ts_tablet_manager.cc:1397] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.005s
I20260805 22:50:57.956840 27376 log.cc:824] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Log is configured to *not* fsync() on all Append() calls
I20260805 22:50:57.961427 27375 tablet_bootstrap.cc:492] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: No bootstrap required, opened a new log
I20260805 22:50:57.962018 27375 ts_tablet_manager.cc:1397] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20260805 22:50:57.963703 27376 tablet_bootstrap.cc:492] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: No bootstrap required, opened a new log
I20260805 22:50:57.964265 27376 ts_tablet_manager.cc:1397] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260805 22:50:57.979360 27375 raft_consensus.cc:348] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } }
I20260805 22:50:57.980975 27375 raft_consensus.cc:374] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:57.980232 27373 raft_consensus.cc:348] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } }
I20260805 22:50:57.981227 27375 raft_consensus.cc:729] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74d9fc31fc24d70ab696c37509470b1, State: Initialized, Role: FOLLOWER
I20260805 22:50:57.981283 27373 raft_consensus.cc:374] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:57.981627 27373 raft_consensus.cc:729] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fff5363cf9449dbfe7cc060b19165a, State: Initialized, Role: FOLLOWER
I20260805 22:50:57.982969 27375 consensus_queue.cc:260] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } }
I20260805 22:50:57.983165 27373 consensus_queue.cc:260] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } }
I20260805 22:50:57.990839 27376 raft_consensus.cc:348] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } }
I20260805 22:50:57.992658 27376 raft_consensus.cc:374] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:50:57.996878 27376 raft_consensus.cc:729] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d417c90743455aa270caa2dfccba13, State: Initialized, Role: FOLLOWER
I20260805 22:50:57.999017 27376 consensus_queue.cc:260] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } }
I20260805 22:50:58.000725 27375 ts_tablet_manager.cc:1428] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: Time spent starting tablet: real 0.039s user 0.025s sys 0.012s
I20260805 22:50:58.001505 27373 ts_tablet_manager.cc:1428] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: Time spent starting tablet: real 0.045s user 0.036s sys 0.008s
I20260805 22:50:58.002038 27345 heartbeater.cc:503] Master 127.4.28.62:34627 was elected leader, sending a full tablet report...
I20260805 22:50:58.010218 27376 ts_tablet_manager.cc:1428] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Time spent starting tablet: real 0.046s user 0.033s sys 0.004s
I20260805 22:50:58.059371 27386 raft_consensus.cc:482] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:50:58.060079 27386 raft_consensus.cc:504] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } }
I20260805 22:50:58.063776 27386 leader_election.cc:288] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c74d9fc31fc24d70ab696c37509470b1 (127.4.28.3:37095), 80fff5363cf9449dbfe7cc060b19165a (127.4.28.2:39975)
I20260805 22:50:58.082387 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.064676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:50455 (local address 127.4.28.3:37095)
0805 22:50:58.065140 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:58.065165 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:58.066502 (+ 1337us) server_negotiation.cc:407] Connection header received
0805 22:50:58.066952 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:58.066958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:58.067075 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:58.067262 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:58.068650 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.070730 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.077093 (+ 6363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.079693 (+ 2600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.080030 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:58.080326 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:58.081772 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0805 22:50:58.082012 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260805 22:50:58.083235 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.064543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:39975 (local address 127.4.28.1:48413)
0805 22:50:58.066297 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:58.066334 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:58.066652 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:58.067281 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:58.067287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:58.067348 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:58.068278 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:58.068319 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.071058 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.071062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:58.076447 (+ 5385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:58.076497 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.080306 (+ 3809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.080338 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:58.081154 (+ 816us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:58.081379 (+ 225us) client_negotiation.cc:769] Sending connection context
0805 22:50:58.081675 (+ 296us) client_negotiation.cc:240] Negotiation successful
0805 22:50:58.082108 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"mutex_wait_us":506,"spinlock_wait_cycles":3968,"thread_start_us":807,"threads_started":1}
I20260805 22:50:58.083529 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.064872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:48413 (local address 127.4.28.2:39975)
0805 22:50:58.065334 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:58.065354 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:58.066585 (+ 1231us) server_negotiation.cc:407] Connection header received
0805 22:50:58.066884 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:58.066902 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:58.067069 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:58.067263 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:50:58.068650 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.070730 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.077106 (+ 6376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.079676 (+ 2570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.080031 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:58.080297 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:58.082839 (+ 2542us) server_negotiation.cc:299] Negotiation successful
0805 22:50:58.083110 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260805 22:50:58.083174 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.064375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:37095 (local address 127.4.28.1:50455)
0805 22:50:58.066297 (+ 1922us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:58.066334 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0805 22:50:58.066744 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:50:58.067281 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:50:58.067287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:50:58.067329 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:50:58.068386 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:58.068411 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.071056 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.071062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:58.076447 (+ 5385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:50:58.076497 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.080347 (+ 3850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.080367 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:50:58.081211 (+ 844us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:58.081375 (+ 164us) client_negotiation.cc:769] Sending connection context
0805 22:50:58.081675 (+ 300us) client_negotiation.cc:240] Negotiation successful
0805 22:50:58.082107 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665,"mutex_wait_us":729,"spinlock_wait_cycles":74880,"thread_start_us":1373,"threads_started":1}
I20260805 22:50:58.086210 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bc23509a0749189e3fe8c7156ff9fa" candidate_uuid: "21d417c90743455aa270caa2dfccba13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74d9fc31fc24d70ab696c37509470b1" is_pre_election: true
I20260805 22:50:58.086210 27051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bc23509a0749189e3fe8c7156ff9fa" candidate_uuid: "21d417c90743455aa270caa2dfccba13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fff5363cf9449dbfe7cc060b19165a" is_pre_election: true
I20260805 22:50:58.087476 27278 raft_consensus.cc:2454] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d417c90743455aa270caa2dfccba13 in term 0.
I20260805 22:50:58.087468 27051 raft_consensus.cc:2454] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d417c90743455aa270caa2dfccba13 in term 0.
I20260805 22:50:58.089389 26749 leader_election.cc:302] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d417c90743455aa270caa2dfccba13, c74d9fc31fc24d70ab696c37509470b1; no voters:
I20260805 22:50:58.090435 27386 raft_consensus.cc:2781] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:50:58.090798 27386 raft_consensus.cc:482] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:50:58.091044 27386 raft_consensus.cc:3037] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:58.097101 27386 raft_consensus.cc:504] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } }
I20260805 22:50:58.098781 27386 leader_election.cc:288] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [CANDIDATE]: Term 1 election: Requested vote from peers c74d9fc31fc24d70ab696c37509470b1 (127.4.28.3:37095), 80fff5363cf9449dbfe7cc060b19165a (127.4.28.2:39975)
I20260805 22:50:58.099824 27051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bc23509a0749189e3fe8c7156ff9fa" candidate_uuid: "21d417c90743455aa270caa2dfccba13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fff5363cf9449dbfe7cc060b19165a"
I20260805 22:50:58.099824 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bc23509a0749189e3fe8c7156ff9fa" candidate_uuid: "21d417c90743455aa270caa2dfccba13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74d9fc31fc24d70ab696c37509470b1"
I20260805 22:50:58.100292 27278 raft_consensus.cc:3037] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:58.100282 27051 raft_consensus.cc:3037] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:50:58.106910 27278 raft_consensus.cc:2454] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d417c90743455aa270caa2dfccba13 in term 1.
I20260805 22:50:58.107007 27051 raft_consensus.cc:2454] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d417c90743455aa270caa2dfccba13 in term 1.
I20260805 22:50:58.108016 26751 leader_election.cc:302] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21d417c90743455aa270caa2dfccba13, 80fff5363cf9449dbfe7cc060b19165a, c74d9fc31fc24d70ab696c37509470b1; no voters:
I20260805 22:50:58.108681 27386 raft_consensus.cc:2781] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:50:58.109467 27386 raft_consensus.cc:686] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 1 LEADER]: Becoming Leader. State: Replica: 21d417c90743455aa270caa2dfccba13, State: Running, Role: LEADER
I20260805 22:50:58.110495 27386 consensus_queue.cc:237] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } }
I20260805 22:50:58.118827 26635 catalog_manager.cc:5495] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21d417c90743455aa270caa2dfccba13 (127.4.28.1). New cstate: current_term: 1 leader_uuid: "21d417c90743455aa270caa2dfccba13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } health_report { overall_health: HEALTHY } } }
I20260805 22:50:58.158416 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.149667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.4.28.1:35985)
0805 22:50:58.149920 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:58.149928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:58.150066 (+ 138us) server_negotiation.cc:407] Connection header received
0805 22:50:58.150337 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:58.150342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:58.150482 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:58.150706 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:50:58.151219 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.152236 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.154631 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.155360 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.155536 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:58.156404 (+ 868us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:50:58.157344 (+ 940us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:50:58.157644 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:58.157781 (+ 137us) server_negotiation.cc:299] Negotiation successful
0805 22:50:58.157937 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260805 22:50:58.172320 27051 raft_consensus.cc:1261] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [term 1 FOLLOWER]: Refusing update from remote peer 21d417c90743455aa270caa2dfccba13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:58.172353 27278 raft_consensus.cc:1261] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [term 1 FOLLOWER]: Refusing update from remote peer 21d417c90743455aa270caa2dfccba13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:50:58.173799 27396 consensus_queue.cc:1035] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:58.174326 27386 consensus_queue.cc:1035] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:50:58.193959 27399 mvcc.cc:205] Tried to move back new op lower bound from 7315334177459429376 to 7315334177228738560. Current Snapshot: MvccSnapshot[applied={T|T < 7315334177459429376}]
I20260805 22:50:58.230247 26635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:50:58.230561 26635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:50:58.234665 26635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Sending DeleteTablet for 3 replicas of tablet 48bc23509a0749189e3fe8c7156ff9fa
I20260805 22:50:58.236060 27025 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bc23509a0749189e3fe8c7156ff9fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:58 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:57252
I20260805 22:50:58.236047 27249 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bc23509a0749189e3fe8c7156ff9fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:58 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:34346
I20260805 22:50:58.236268 26824 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bc23509a0749189e3fe8c7156ff9fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:50:58 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:36620
I20260805 22:50:58.236501 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 26727
I20260805 22:50:58.251425 27411 tablet_replica.cc:330] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: stopping tablet replica
I20260805 22:50:58.252331 27411 raft_consensus.cc:2229] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:50:58.253144 27411 raft_consensus.cc:2258] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:58.256237 27411 ts_tablet_manager.cc:1905] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:58.256315 27415 tablet_replica.cc:330] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: stopping tablet replica
I20260805 22:50:58.256808 27415 raft_consensus.cc:2229] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:50:58.257397 27415 raft_consensus.cc:2258] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 05 22:50:58 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:50:58.264536 27415 ts_tablet_manager.cc:1905] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:58.266034 27416 tablet_replica.cc:330] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: stopping tablet replica
I20260805 22:50:58.266222 27411 ts_tablet_manager.cc:1918] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:58.266506 27411 log.cc:1196] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/48bc23509a0749189e3fe8c7156ff9fa
I20260805 22:50:58.266553 27416 raft_consensus.cc:2229] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:50:58.267199 27411 ts_tablet_manager.cc:1939] T 48bc23509a0749189e3fe8c7156ff9fa P c74d9fc31fc24d70ab696c37509470b1: Deleting consensus metadata
I20260805 22:50:58.267334 27416 raft_consensus.cc:2258] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:50:58.269541 26611 catalog_manager.cc:4841] TS c74d9fc31fc24d70ab696c37509470b1 (127.4.28.3:37095): tablet 48bc23509a0749189e3fe8c7156ff9fa (table test-table [id=4ded93f334ce45c6b1f470d42c9416e8]) successfully deleted
I20260805 22:50:58.270663 27416 ts_tablet_manager.cc:1905] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:50:58.272794 27415 ts_tablet_manager.cc:1918] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:58.273012 27415 log.cc:1196] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/48bc23509a0749189e3fe8c7156ff9fa
I20260805 22:50:58.273613 27415 ts_tablet_manager.cc:1939] T 48bc23509a0749189e3fe8c7156ff9fa P 80fff5363cf9449dbfe7cc060b19165a: Deleting consensus metadata
I20260805 22:50:58.275758 26613 catalog_manager.cc:4841] TS 80fff5363cf9449dbfe7cc060b19165a (127.4.28.2:39975): tablet 48bc23509a0749189e3fe8c7156ff9fa (table test-table [id=4ded93f334ce45c6b1f470d42c9416e8]) successfully deleted
I20260805 22:50:58.276067 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.237966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.4.28.1:35985)
0805 22:50:58.238207 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:58.238214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:58.250420 (+ 12206us) server_negotiation.cc:407] Connection header received
0805 22:50:58.251050 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:58.251058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:58.251153 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:58.251239 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:58.253687 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.255070 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.257324 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.258092 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.258205 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:58.267270 (+ 9065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:58.267292 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:58.267313 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:58.267379 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:58.271627 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:58.272829 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:58.272841 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:58.272844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:58.273105 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:58.273806 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:58.273812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:58.273815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:58.274355 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:58.274675 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:58.275390 (+ 715us) server_negotiation.cc:299] Negotiation successful
0805 22:50:58.275687 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260805 22:50:58.277585 26911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:58.377368 27416 ts_tablet_manager.cc:1918] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:50:58.377724 27416 log.cc:1196] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/48bc23509a0749189e3fe8c7156ff9fa
I20260805 22:50:58.378074 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 26727
I20260805 22:50:58.401335 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 26927
W20260805 22:50:58.401417 26616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.28.1:35985 (error 108)
I20260805 22:50:58.403421 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.402705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.0.0.1:36646)
0805 22:50:58.403120 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:58.403244 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:35985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":197,"threads_started":1}
W20260805 22:50:58.403669 26616 catalog_manager.cc:4584] TS 21d417c90743455aa270caa2dfccba13 (127.4.28.1:35985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48bc23509a0749189e3fe8c7156ff9fa: Network error: Client connection negotiation failed: client connection to 127.4.28.1:35985: connect: Connection refused (error 111)
Aug 05 22:50:58 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:50:58.418784 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.402259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57268 (local address 127.4.28.2:39975)
0805 22:50:58.402459 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:58.402466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:58.402608 (+ 142us) server_negotiation.cc:407] Connection header received
0805 22:50:58.402779 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:58.402784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:58.402897 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:58.403019 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:58.404795 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.405729 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.407321 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.407756 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.407864 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:58.412596 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:58.412614 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:58.412634 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:58.412684 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:58.415678 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:58.416588 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:58.416597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:58.416599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:58.416778 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:58.417206 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:58.417209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:58.417211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:58.417698 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:58.417976 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:58.418269 (+ 293us) server_negotiation.cc:299] Negotiation successful
0805 22:50:58.418460 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 22:50:58.419519 27119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:58.457473 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.456968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.0.0.1:36652)
0805 22:50:58.457211 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:58.457305 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:35985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260805 22:50:58.458945 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.458422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.0.0.1:36654)
0805 22:50:58.458628 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:58.458716 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:35985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260805 22:50:58.512029 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 26927
I20260805 22:50:58.519848 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.519269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.0.0.1:36666)
0805 22:50:58.519538 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:58.519670 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:35985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260805 22:50:58.521342 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.520689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.0.0.1:36670)
0805 22:50:58.521010 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:58.521115 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:35985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260805 22:50:58.540637 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 27136
Aug 05 22:50:58 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:50:58.557678 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.541551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.4.28.3:37095)
0805 22:50:58.541744 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 22:50:58.541751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:50:58.541772 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:50:58.541961 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:50:58.541967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:50:58.542044 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:50:58.542129 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:50:58.543545 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.544270 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.545992 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:50:58.546405 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:50:58.546524 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:50:58.551375 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:50:58.551397 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:50:58.551417 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:50:58.551469 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:50:58.554693 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:58.555546 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:58.555556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:58.555559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:58.555727 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:50:58.556152 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:50:58.556157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:50:58.556159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:50:58.556590 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:50:58.556893 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0805 22:50:58.557135 (+ 242us) server_negotiation.cc:299] Negotiation successful
0805 22:50:58.557335 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260805 22:50:58.558365 27339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:58.602082 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.601496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.0.0.1:36672)
0805 22:50:58.601813 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:58.601925 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:35985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260805 22:50:58.603381 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:50:58.602924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.0.0.1:36680)
0805 22:50:58.603136 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0805 22:50:58.603211 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:35985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260805 22:50:58.650143 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 27136
I20260805 22:50:58.674396 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 26588
I20260805 22:50:58.675736 26679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:50:58.791795 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 26588
I20260805 22:51:18.820529 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34627
--webserver_interface=127.4.28.62
--webserver_port=41965
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:34627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:51:19.185709 27430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:51:19.186053 27430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:51:19.186190 27430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:51:19.193725 27430 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260805 22:51:19.193840 27430 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260805 22:51:19.193900 27430 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260805 22:51:19.193981 27430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:51:19.194054 27430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:51:19.194147 27430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:51:19.194211 27430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:51:19.194289 27430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:51:19.204718 27430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:34627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:51:19.206249 27430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:51:19.207670 27430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:51:19.235780 27441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:19.235788 27438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:19.235765 27439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:51:19.236492 27430 server_base.cc:1034] running on GCE node
I20260805 22:51:19.237726 27430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:51:19.239646 27430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:51:19.240954 27430 hybrid_clock.cc:648] HybridClock initialized: now 1785970279240856 us; error 61 us; skew 500 ppm
I20260805 22:51:19.241425 27430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:51:19 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970279, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:51:19.245661 27430 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:51:19.247522 27430 webserver.cc:460] Webserver started at http://127.4.28.62:41965/ using document root <none> and password file <none>
I20260805 22:51:19.248443 27430 fs_manager.cc:359] Metadata directory not provided
I20260805 22:51:19.248592 27430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:51:19.255577 27430 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 22:51:19.258910 27452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:19.260586 27430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:51:19.260797 27430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f"
format_stamp: "Formatted at 2026-08-05 22:50:54 on dist-test-slave-7lbf"
server_key: "bf8dab32a3208b2f9e3f1046e3deff4a"
server_key_iv: "b36c7465511365d2d119de166568bcba"
server_key_version: "encryptionkey@0"
I20260805 22:51:19.262014 27430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:51:19.275753 27430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:51:19.280248 27430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:51:19.280553 27430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:51:19.299968 27430 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:34627
I20260805 22:51:19.299993 27531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:34627 every 8 connection(s)
I20260805 22:51:19.301705 27430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:51:19.305012 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 27430
I20260805 22:51:19.305935 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:35985
--local_ip_for_outbound_sockets=127.4.28.1
--tserver_master_addrs=127.4.28.62:34627
--webserver_port=42595
--webserver_interface=127.4.28.1
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:51:19.309754 27532 sys_catalog.cc:263] Verifying existing consensus state
I20260805 22:51:19.313402 27532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Bootstrap starting.
I20260805 22:51:19.338953 27532 log.cc:824] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Log is configured to *not* fsync() on all Append() calls
I20260805 22:51:19.356915 27532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:51:19.357733 27532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Bootstrap complete.
I20260805 22:51:19.371959 27532 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:51:19.373337 27532 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fdc5ff3fcd48378a29acfdc0f2710f, State: Initialized, Role: FOLLOWER
I20260805 22:51:19.374233 27532 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:51:19.374567 27532 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:51:19.374723 27532 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:51:19.374931 27532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:51:19.381248 27532 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:51:19.382086 27532 leader_election.cc:302] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1fdc5ff3fcd48378a29acfdc0f2710f; no voters:
I20260805 22:51:19.382817 27532 leader_election.cc:288] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260805 22:51:19.383024 27536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:51:19.384169 27536 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 2 LEADER]: Becoming Leader. State: Replica: a1fdc5ff3fcd48378a29acfdc0f2710f, State: Running, Role: LEADER
I20260805 22:51:19.384738 27536 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:51:19.385239 27532 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:51:19.387392 27538 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fdc5ff3fcd48378a29acfdc0f2710f. Latest consensus state: current_term: 2 leader_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } } }
I20260805 22:51:19.387392 27537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } } }
I20260805 22:51:19.387823 27537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: This master's current role is: LEADER
I20260805 22:51:19.387826 27538 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: This master's current role is: LEADER
I20260805 22:51:19.388720 27544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:51:19.401167 27544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:51:19.402904 27544 catalog_manager.cc:1259] Loaded cluster ID: 729745043b6342dcba34024a453b8f3e
I20260805 22:51:19.403131 27544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:51:19.433897 27544 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:51:19.435842 27544 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Loaded TSK: 0
I20260805 22:51:19.457926 27559 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260805 22:51:19.486575 27544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Generated new TSK 1
I20260805 22:51:19.487185 27544 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 22:51:19.738591 27534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:51:19.738925 27534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:51:19.739055 27534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:51:19.746749 27534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:51:19.746881 27534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:51:19.747035 27534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:51:19.758569 27534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:35985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=42595
--tserver_master_addrs=127.4.28.62:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:51:19.760052 27534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:51:19.761556 27534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:51:19.782692 27571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:19.782843 27566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:19.783123 27567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:51:19.783743 27534 server_base.cc:1034] running on GCE node
I20260805 22:51:19.784677 27534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:51:19.786171 27534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:51:19.787456 27534 hybrid_clock.cc:648] HybridClock initialized: now 1785970279787353 us; error 108 us; skew 500 ppm
I20260805 22:51:19.787897 27534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:51:19 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970279, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:51:19.792636 27534 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:51:19.794871 27534 webserver.cc:460] Webserver started at http://127.4.28.1:42595/ using document root <none> and password file <none>
I20260805 22:51:19.795971 27534 fs_manager.cc:359] Metadata directory not provided
I20260805 22:51:19.796192 27534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:51:19.804554 27534 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260805 22:51:19.808611 27582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:19.810577 27534 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:51:19.810796 27534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "21d417c90743455aa270caa2dfccba13"
format_stamp: "Formatted at 2026-08-05 22:50:55 on dist-test-slave-7lbf"
server_key: "8848b9239fe5bbf32360cbbef7697da6"
server_key_iv: "8c596d9d3e988ff94e652c4578fb5522"
server_key_version: "encryptionkey@0"
I20260805 22:51:19.812300 27534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:51:19.835736 27534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:51:19.841670 27534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:51:19.842003 27534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:51:19.843328 27534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:51:19.845832 27592 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260805 22:51:19.851956 27592 ts_tablet_manager.cc:1852] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260805 22:51:19.853010 27592 ts_tablet_manager.cc:1905] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:51:19.861272 27592 ts_tablet_manager.cc:1918] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:51:19.861649 27592 ts_tablet_manager.cc:1939] T 48bc23509a0749189e3fe8c7156ff9fa P 21d417c90743455aa270caa2dfccba13: Deleting consensus metadata
I20260805 22:51:19.862121 27534 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260805 22:51:19.862254 27534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.001s
I20260805 22:51:19.862387 27534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:51:19.862459 27534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:19.897603 27534 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:35985
I20260805 22:51:19.897663 27741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:35985 every 8 connection(s)
I20260805 22:51:19.899031 27534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:51:19.909237 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 27534
I20260805 22:51:19.910122 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:39975
--local_ip_for_outbound_sockets=127.4.28.2
--tserver_master_addrs=127.4.28.62:34627
--webserver_port=41449
--webserver_interface=127.4.28.2
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:51:19 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970279, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:51:19.925839 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:19.902071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:43889 (local address 127.4.28.62:34627)
0805 22:51:19.902649 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:19.902662 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:19.903291 (+ 629us) server_negotiation.cc:407] Connection header received
0805 22:51:19.905675 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:19.905692 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:19.906090 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:19.906419 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:19.907979 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:19.909856 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:19.911711 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:19.912748 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:19.912952 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:19.916936 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:19.916952 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:19.916977 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:19.917047 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:19.920859 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:19.921899 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:19.921911 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:19.921920 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:19.922186 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:19.922658 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:19.922663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:19.922666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:19.923346 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:19.923543 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:19.924395 (+ 852us) server_negotiation.cc:299] Negotiation successful
0805 22:51:19.924782 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":240,"threads_started":1}
I20260805 22:51:19.926141 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:19.902237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:43889)
0805 22:51:19.903106 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:19.903154 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:19.905071 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:19.906601 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:19.906609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:19.907165 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:51:19.907784 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:19.907805 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:19.910057 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:19.910062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:19.911519 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:19.911536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:19.913005 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:19.913011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:19.913210 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:19.913962 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:51:19.913989 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:51:19.916529 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:51:19.921067 (+ 4538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:51:19.921077 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:51:19.921092 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:51:19.921713 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:51:19.922311 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:51:19.922314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:51:19.922317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:51:19.922533 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:51:19.923475 (+ 942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:51:19.923482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:51:19.923833 (+ 351us) client_negotiation.cc:769] Sending connection context
0805 22:51:19.924096 (+ 263us) client_negotiation.cc:240] Negotiation successful
0805 22:51:19.924644 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":357,"threads_started":1}
I20260805 22:51:19.929828 27742 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34627
I20260805 22:51:19.930179 27742 heartbeater.cc:464] Registering TS with master...
I20260805 22:51:19.930984 27742 heartbeater.cc:511] Master 127.4.28.62:34627 requested a full tablet report, sending...
I20260805 22:51:19.933672 27474 ts_manager.cc:194] Registered new tserver with Master: 21d417c90743455aa270caa2dfccba13 (127.4.28.1:35985)
I20260805 22:51:19.935878 27474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:43889
W20260805 22:51:20.282893 27747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:51:20.283219 27747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:51:20.283329 27747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:51:20.290934 27747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:51:20.291064 27747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:51:20.291190 27747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:51:20.301493 27747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:39975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=41449
--tserver_master_addrs=127.4.28.62:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:51:20.302949 27747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:51:20.304317 27747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:51:20.318790 27754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:20.318764 27757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:20.319139 27759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:51:20.319772 27747 server_base.cc:1034] running on GCE node
I20260805 22:51:20.320493 27747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:51:20.321614 27747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:51:20.322866 27747 hybrid_clock.cc:648] HybridClock initialized: now 1785970280322803 us; error 54 us; skew 500 ppm
I20260805 22:51:20.323246 27747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:51:20 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970280, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:51:20.326764 27747 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:51:20.328598 27747 webserver.cc:460] Webserver started at http://127.4.28.2:41449/ using document root <none> and password file <none>
I20260805 22:51:20.329489 27747 fs_manager.cc:359] Metadata directory not provided
I20260805 22:51:20.329703 27747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:51:20.336093 27747 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:51:20.339375 27768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:20.340952 27747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 22:51:20.341154 27747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "80fff5363cf9449dbfe7cc060b19165a"
format_stamp: "Formatted at 2026-08-05 22:50:56 on dist-test-slave-7lbf"
server_key: "670640efc9bb08f70efc62ca17e175b9"
server_key_iv: "c8880aa68abe617736e2384929f1c4da"
server_key_version: "encryptionkey@0"
I20260805 22:51:20.342278 27747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:51:20.376068 27747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:51:20.381191 27747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:51:20.381500 27747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:51:20.382635 27747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:51:20.384181 27747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:51:20.384310 27747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:20.384434 27747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:51:20.384509 27747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:20.417972 27747 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:39975
I20260805 22:51:20.418025 27927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:39975 every 8 connection(s)
I20260805 22:51:20.419552 27747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:51:20.420274 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 27747
I20260805 22:51:20.421136 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:37095
--local_ip_for_outbound_sockets=127.4.28.3
--tserver_master_addrs=127.4.28.62:34627
--webserver_port=45157
--webserver_interface=127.4.28.3
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 22:51:20 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970280, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:51:20.442406 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.422418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:36273 (local address 127.4.28.62:34627)
0805 22:51:20.422598 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:20.422603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:20.424202 (+ 1599us) server_negotiation.cc:407] Connection header received
0805 22:51:20.426176 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:20.426182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:20.426295 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:20.426386 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:20.428045 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.428795 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.430574 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.431048 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.431201 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.435397 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:20.435402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:20.435406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:20.435446 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:20.438843 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:20.439787 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:20.439793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:20.439797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:20.439997 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:20.440459 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:20.440464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:20.440467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:20.440839 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:20.440983 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:20.441803 (+ 820us) server_negotiation.cc:299] Negotiation successful
0805 22:51:20.442067 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 22:51:20.443331 27930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.423160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:36273)
0805 22:51:20.424008 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:20.424060 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:20.425868 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:20.426668 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:20.426677 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:20.427245 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:51:20.427846 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.427865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.428992 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.428995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:20.430382 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.430397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.431343 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.431351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:20.431663 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.432409 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:51:20.432439 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:51:20.435096 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:51:20.439057 (+ 3961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:51:20.439067 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:51:20.439084 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:51:20.439613 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:51:20.440108 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:51:20.440112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:51:20.440114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:51:20.440334 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:51:20.440975 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:51:20.440981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:51:20.441360 (+ 379us) client_negotiation.cc:769] Sending connection context
0805 22:51:20.441666 (+ 306us) client_negotiation.cc:240] Negotiation successful
0805 22:51:20.442126 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":342,"threads_started":1}
I20260805 22:51:20.445040 27928 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34627
I20260805 22:51:20.445421 27928 heartbeater.cc:464] Registering TS with master...
I20260805 22:51:20.446478 27928 heartbeater.cc:511] Master 127.4.28.62:34627 requested a full tablet report, sending...
I20260805 22:51:20.448280 27474 ts_manager.cc:194] Registered new tserver with Master: 80fff5363cf9449dbfe7cc060b19165a (127.4.28.2:39975)
I20260805 22:51:20.449208 27474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:36273
W20260805 22:51:20.783687 27931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:51:20.784017 27931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:51:20.784142 27931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:51:20.794337 27931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:51:20.794476 27931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:51:20.794646 27931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:51:20.805037 27931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:37095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=45157
--tserver_master_addrs=127.4.28.62:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:51:20.806463 27931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:51:20.807888 27931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:51:20.821588 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:20.821697 27937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:20.821676 27936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:51:20.822757 27931 server_base.cc:1034] running on GCE node
I20260805 22:51:20.823423 27931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:51:20.824473 27931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:51:20.825695 27931 hybrid_clock.cc:648] HybridClock initialized: now 1785970280825601 us; error 98 us; skew 500 ppm
I20260805 22:51:20.826054 27931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:51:20 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970280, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:51:20.829341 27931 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:51:20.831169 27931 webserver.cc:460] Webserver started at http://127.4.28.3:45157/ using document root <none> and password file <none>
I20260805 22:51:20.831887 27931 fs_manager.cc:359] Metadata directory not provided
I20260805 22:51:20.832018 27931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:51:20.838069 27931 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260805 22:51:20.841264 27948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:20.842906 27931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:51:20.843092 27931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "c74d9fc31fc24d70ab696c37509470b1"
format_stamp: "Formatted at 2026-08-05 22:50:57 on dist-test-slave-7lbf"
server_key: "a6b5928e0b7acb5a250357ae9ac11be7"
server_key_iv: "4dbe66b65414a8a5858099a3fcf3b012"
server_key_version: "encryptionkey@0"
I20260805 22:51:20.844134 27931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:51:20.861732 27931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:51:20.865949 27931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:51:20.866199 27931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:51:20.867228 27931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:51:20.868716 27931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:51:20.868817 27931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:20.868929 27931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:51:20.869005 27931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:20.903532 27931 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:37095
I20260805 22:51:20.903594 28112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:37095 every 8 connection(s)
I20260805 22:51:20.905004 27931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:51:20.909382 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 27931
Aug 05 22:51:20 dist-test-slave-7lbf krb5kdc[26557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970280, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:51:20.925076 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.910328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.4.28.62:34627)
0805 22:51:20.910833 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:20.910839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:20.911145 (+ 306us) server_negotiation.cc:407] Connection header received
0805 22:51:20.911364 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:20.911368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:20.911437 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:20.911535 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:20.913176 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.914147 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.915566 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.916152 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.916268 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.919292 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:20.919298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:20.919301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:20.919335 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:20.922296 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:20.923041 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:20.923046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:20.923049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:20.923173 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:20.923584 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:20.923589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:20.923591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:20.923863 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:20.924116 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:20.924458 (+ 342us) server_negotiation.cc:299] Negotiation successful
0805 22:51:20.924685 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":385,"spinlock_wait_cycles":23808,"thread_start_us":246,"threads_started":1}
I20260805 22:51:20.926756 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.908172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:49035)
0805 22:51:20.908980 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:20.909030 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:20.911164 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:20.911890 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:20.911900 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:20.912453 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:51:20.913066 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.913085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.914254 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.914258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:20.915541 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.915554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.916458 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.916467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:20.916715 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.917317 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:51:20.917342 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:51:20.919636 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:51:20.922842 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:51:20.922852 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:51:20.922868 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:51:20.923290 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:51:20.923637 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:51:20.923642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:51:20.923646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:51:20.923848 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:51:20.924367 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:51:20.924375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:51:20.924712 (+ 337us) client_negotiation.cc:769] Sending connection context
0805 22:51:20.924986 (+ 274us) client_negotiation.cc:240] Negotiation successful
0805 22:51:20.925540 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":292,"threads_started":1}
I20260805 22:51:20.927701 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.907738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:49035 (local address 127.4.28.62:34627)
0805 22:51:20.907942 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:20.907947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:20.909142 (+ 1195us) server_negotiation.cc:407] Connection header received
0805 22:51:20.911440 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:20.911445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:20.911522 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:20.911605 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:20.913238 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.914081 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.915687 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.916225 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.916347 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.919841 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:20.919846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:20.919849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:20.919880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:20.922692 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:20.923411 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:20.923417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:20.923421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:20.923530 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:20.923958 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:20.923963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:20.923966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:20.924240 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:20.924375 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:20.927234 (+ 2859us) server_negotiation.cc:299] Negotiation successful
0805 22:51:20.927425 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":69,"spinlock_wait_cycles":768}
I20260805 22:51:20.928711 28113 heartbeater.cc:347] Connected to a master server at 127.4.28.62:34627
I20260805 22:51:20.928997 28113 heartbeater.cc:464] Registering TS with master...
I20260805 22:51:20.929778 28113 heartbeater.cc:511] Master 127.4.28.62:34627 requested a full tablet report, sending...
I20260805 22:51:20.931274 27474 ts_manager.cc:194] Registered new tserver with Master: c74d9fc31fc24d70ab696c37509470b1 (127.4.28.3:37095)
I20260805 22:51:20.932094 27474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:49035
I20260805 22:51:20.939714 27742 heartbeater.cc:503] Master 127.4.28.62:34627 was elected leader, sending a full tablet report...
I20260805 22:51:20.940301 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:51:20.947440 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.941202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.4.28.62:34627)
0805 22:51:20.941397 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:20.941403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:20.941923 (+ 520us) server_negotiation.cc:407] Connection header received
0805 22:51:20.942095 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:20.942099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:20.942236 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:20.942364 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:51:20.942766 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.943525 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.945248 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.945698 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.945821 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.946429 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:51:20.946760 (+ 331us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0805 22:51:20.947161 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57372: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260805 22:51:20.947585 27744 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57372: token expired
W20260805 22:51:20.948295 27354 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.28.62:34627)
I20260805 22:51:20.961833 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.948977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.4.28.62:34627)
0805 22:51:20.949142 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:20.949146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:20.949162 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:51:20.949337 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:20.949341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:20.949408 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:20.949498 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:20.950932 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.951766 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.953184 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.953653 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.953750 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.956438 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:20.956443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:20.956447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:20.956488 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:20.959010 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:20.959806 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:20.959811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:20.959814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:20.959973 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:20.960345 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:20.960349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:20.960351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:20.960740 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:20.961010 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:20.961263 (+ 253us) server_negotiation.cc:299] Negotiation successful
0805 22:51:20.961465 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 22:51:20.965950 27474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:51:20.974231 27474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:51:21.000337 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.987408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.4.28.2:39975)
0805 22:51:20.988724 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:20.988734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:20.990969 (+ 2235us) server_negotiation.cc:407] Connection header received
0805 22:51:20.991311 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:20.991322 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:20.991600 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:20.991796 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:51:20.992902 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.994435 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.997465 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.998705 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.998848 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.999067 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:20.999778 (+ 711us) server_negotiation.cc:299] Negotiation successful
0805 22:51:20.999978 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":1037,"threads_started":1}
I20260805 22:51:21.000475 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.986912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.0.0.1:38880)
0805 22:51:20.989323 (+ 2411us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:20.989377 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:20.989596 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:20.993281 (+ 3685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:20.993286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:20.993321 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:51:20.993736 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.993749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.995666 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.995672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:20.997493 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.997508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.999590 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.999596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:20.999761 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.999792 (+ 31us) client_negotiation.cc:769] Sending connection context
0805 22:51:20.999872 (+ 80us) client_negotiation.cc:240] Negotiation successful
0805 22:51:21.000073 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2134,"mutex_wait_us":59,"thread_start_us":2244,"threads_started":1}
I20260805 22:51:21.001116 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.987222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:39975 (local address 127.0.0.1:54474)
0805 22:51:20.990839 (+ 3617us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:20.990857 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:20.991075 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:20.991919 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:20.991928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:20.991959 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:51:20.992670 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.992688 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.994643 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.994649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:20.997263 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.997284 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.999246 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.999257 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:20.999456 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.999518 (+ 62us) client_negotiation.cc:769] Sending connection context
0805 22:51:21.000635 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0805 22:51:21.000776 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3493,"thread_start_us":4969,"threads_started":1}
I20260805 22:51:21.001308 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.987174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.4.28.1:35985)
0805 22:51:20.992719 (+ 5545us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:20.992728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:20.992758 (+ 30us) server_negotiation.cc:407] Connection header received
0805 22:51:20.992871 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:20.992879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:20.993097 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:20.993272 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:51:20.993984 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.995514 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.997681 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.999068 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.999196 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:20.999429 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.000897 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.001038 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5367,"thread_start_us":5248,"threads_started":1}
I20260805 22:51:21.001315 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.989369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.4.28.3:37095)
0805 22:51:20.990375 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:20.990384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:20.992801 (+ 2417us) server_negotiation.cc:407] Connection header received
0805 22:51:20.992961 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:20.992969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:20.993233 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:20.993444 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:51:20.994242 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.995821 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.998436 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.999846 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.999964 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.000223 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.000791 (+ 568us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.000964 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":619,"threads_started":1}
I20260805 22:51:21.002024 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:20.988969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:37095 (local address 127.0.0.1:48416)
0805 22:51:20.992648 (+ 3679us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:20.992668 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:20.992808 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:20.993615 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:20.993623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:20.993662 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:51:20.994024 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.994034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:20.995978 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:20.995984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:20.998244 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:20.998261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.000402 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.000408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:21.000581 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.000614 (+ 33us) client_negotiation.cc:769] Sending connection context
0805 22:51:21.001508 (+ 894us) client_negotiation.cc:240] Negotiation successful
0805 22:51:21.001675 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3512,"mutex_wait_us":27,"thread_start_us":2581,"threads_started":1}
I20260805 22:51:21.003602 27645 tablet_service.cc:1467] Processing CreateTablet for tablet 21fe6166c9114186997627744aa7f2f2 (DEFAULT_TABLE table=test-table [id=25377c25546d4baca6b79c49a8020b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:51:21.005242 27645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fe6166c9114186997627744aa7f2f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:51:21.005237 28012 tablet_service.cc:1467] Processing CreateTablet for tablet 21fe6166c9114186997627744aa7f2f2 (DEFAULT_TABLE table=test-table [id=25377c25546d4baca6b79c49a8020b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:51:21.005314 27828 tablet_service.cc:1467] Processing CreateTablet for tablet 21fe6166c9114186997627744aa7f2f2 (DEFAULT_TABLE table=test-table [id=25377c25546d4baca6b79c49a8020b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:51:21.006711 28012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fe6166c9114186997627744aa7f2f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:51:21.006796 27828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fe6166c9114186997627744aa7f2f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:51:21.017428 28134 tablet_bootstrap.cc:492] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: Bootstrap starting.
I20260805 22:51:21.021634 28135 tablet_bootstrap.cc:492] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: Bootstrap starting.
I20260805 22:51:21.022795 28134 tablet_bootstrap.cc:654] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: Neither blocks nor log segments found. Creating new log.
I20260805 22:51:21.023386 28136 tablet_bootstrap.cc:492] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: Bootstrap starting.
I20260805 22:51:21.024402 28134 log.cc:824] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: Log is configured to *not* fsync() on all Append() calls
I20260805 22:51:21.025530 28135 tablet_bootstrap.cc:654] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: Neither blocks nor log segments found. Creating new log.
I20260805 22:51:21.026705 28135 log.cc:824] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: Log is configured to *not* fsync() on all Append() calls
I20260805 22:51:21.027084 28136 tablet_bootstrap.cc:654] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: Neither blocks nor log segments found. Creating new log.
I20260805 22:51:21.028162 28134 tablet_bootstrap.cc:492] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: No bootstrap required, opened a new log
I20260805 22:51:21.028474 28134 ts_tablet_manager.cc:1397] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260805 22:51:21.028447 28136 log.cc:824] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: Log is configured to *not* fsync() on all Append() calls
I20260805 22:51:21.029784 28135 tablet_bootstrap.cc:492] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: No bootstrap required, opened a new log
I20260805 22:51:21.030233 28135 ts_tablet_manager.cc:1397] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260805 22:51:21.032322 28136 tablet_bootstrap.cc:492] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: No bootstrap required, opened a new log
I20260805 22:51:21.032629 28136 ts_tablet_manager.cc:1397] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260805 22:51:21.037649 28134 raft_consensus.cc:348] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.038069 28134 raft_consensus.cc:374] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:51:21.038211 28134 raft_consensus.cc:729] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d417c90743455aa270caa2dfccba13, State: Initialized, Role: FOLLOWER
I20260805 22:51:21.038861 28134 consensus_queue.cc:260] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.041157 28135 raft_consensus.cc:348] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.041733 28135 raft_consensus.cc:374] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:51:21.041882 28135 raft_consensus.cc:729] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fff5363cf9449dbfe7cc060b19165a, State: Initialized, Role: FOLLOWER
I20260805 22:51:21.042606 28134 ts_tablet_manager.cc:1428] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260805 22:51:21.042697 28135 consensus_queue.cc:260] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.043277 28136 raft_consensus.cc:348] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.043839 28136 raft_consensus.cc:374] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:51:21.043990 28136 raft_consensus.cc:729] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74d9fc31fc24d70ab696c37509470b1, State: Initialized, Role: FOLLOWER
I20260805 22:51:21.044052 27928 heartbeater.cc:503] Master 127.4.28.62:34627 was elected leader, sending a full tablet report...
I20260805 22:51:21.044762 28135 ts_tablet_manager.cc:1428] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260805 22:51:21.044829 28136 consensus_queue.cc:260] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.046213 28113 heartbeater.cc:503] Master 127.4.28.62:34627 was elected leader, sending a full tablet report...
I20260805 22:51:21.046806 28136 ts_tablet_manager.cc:1428] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
W20260805 22:51:21.151535 27743 tablet.cc:2369] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:51:21.156646 28114 tablet.cc:2369] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:51:21.171922 27929 tablet.cc:2369] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:51:21.219955 28142 raft_consensus.cc:482] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:51:21.220336 28142 raft_consensus.cc:504] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.222229 28142 leader_election.cc:288] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21d417c90743455aa270caa2dfccba13 (127.4.28.1:35985), c74d9fc31fc24d70ab696c37509470b1 (127.4.28.3:37095)
I20260805 22:51:21.224244 28141 raft_consensus.cc:482] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:51:21.224514 28141 raft_consensus.cc:504] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.226644 28141 leader_election.cc:288] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fff5363cf9449dbfe7cc060b19165a (127.4.28.2:39975), c74d9fc31fc24d70ab696c37509470b1 (127.4.28.3:37095)
I20260805 22:51:21.230566 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.222715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:57421 (local address 127.4.28.3:37095)
0805 22:51:21.222953 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:21.222960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:21.223724 (+ 764us) server_negotiation.cc:407] Connection header received
0805 22:51:21.223960 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:21.223965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:21.224043 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:21.224129 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:51:21.224690 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.225647 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.227847 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.229143 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.229246 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.229360 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.230067 (+ 707us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.230227 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260805 22:51:21.231958 28144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.222722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:37095 (local address 127.4.28.2:57421)
0805 22:51:21.223601 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:21.223620 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:21.223809 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:21.224203 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:21.224208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:21.224244 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:51:21.224543 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:21.224554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.225779 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.225785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:21.227619 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:21.227639 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.229661 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.229669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:21.229857 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.229894 (+ 37us) client_negotiation.cc:769] Sending connection context
0805 22:51:21.231219 (+ 1325us) client_negotiation.cc:240] Negotiation successful
0805 22:51:21.231450 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"spinlock_wait_cycles":29056,"thread_start_us":469,"threads_started":1}
I20260805 22:51:21.232666 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.222788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:49595 (local address 127.4.28.1:35985)
0805 22:51:21.223012 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:21.223020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:21.223679 (+ 659us) server_negotiation.cc:407] Connection header received
0805 22:51:21.223876 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:21.223880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:21.223943 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:21.224023 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:51:21.225237 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.226383 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.228886 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.230290 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.230396 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.230504 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.232229 (+ 1725us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.232395 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260805 22:51:21.233495 28045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fe6166c9114186997627744aa7f2f2" candidate_uuid: "80fff5363cf9449dbfe7cc060b19165a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74d9fc31fc24d70ab696c37509470b1" is_pre_election: true
I20260805 22:51:21.234156 28045 raft_consensus.cc:2454] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fff5363cf9449dbfe7cc060b19165a in term 0.
I20260805 22:51:21.234895 28145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.222660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35985 (local address 127.4.28.2:49595)
0805 22:51:21.223558 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:21.223575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:21.223739 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:21.224063 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:21.224067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:21.224685 (+ 618us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:51:21.225050 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:21.225062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.226572 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.226577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:21.228669 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:21.228688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.231794 (+ 3106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.231801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:21.232026 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.232059 (+ 33us) client_negotiation.cc:769] Sending connection context
0805 22:51:21.234360 (+ 2301us) client_negotiation.cc:240] Negotiation successful
0805 22:51:21.234518 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":517,"threads_started":1}
I20260805 22:51:21.235090 27769 leader_election.cc:302] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80fff5363cf9449dbfe7cc060b19165a, c74d9fc31fc24d70ab696c37509470b1; no voters:
I20260805 22:51:21.235638 28142 raft_consensus.cc:2781] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:51:21.235793 28142 raft_consensus.cc:482] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:51:21.235901 28142 raft_consensus.cc:3037] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:51:21.235986 27683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fe6166c9114186997627744aa7f2f2" candidate_uuid: "80fff5363cf9449dbfe7cc060b19165a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d417c90743455aa270caa2dfccba13" is_pre_election: true
I20260805 22:51:21.236505 27683 raft_consensus.cc:2454] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fff5363cf9449dbfe7cc060b19165a in term 0.
I20260805 22:51:21.238950 28142 raft_consensus.cc:504] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.240310 27683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fe6166c9114186997627744aa7f2f2" candidate_uuid: "80fff5363cf9449dbfe7cc060b19165a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d417c90743455aa270caa2dfccba13"
I20260805 22:51:21.240440 28045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fe6166c9114186997627744aa7f2f2" candidate_uuid: "80fff5363cf9449dbfe7cc060b19165a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74d9fc31fc24d70ab696c37509470b1"
I20260805 22:51:21.240558 27683 raft_consensus.cc:3037] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:51:21.240705 28045 raft_consensus.cc:3037] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:51:21.241427 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.227474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:38469 (local address 127.4.28.2:39975)
0805 22:51:21.227676 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:21.227682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:21.233887 (+ 6205us) server_negotiation.cc:407] Connection header received
0805 22:51:21.234071 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:21.234077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:21.234152 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:21.234234 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:51:21.234983 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.236063 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.238829 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.240221 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.240357 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.240455 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.241041 (+ 586us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.241185 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 22:51:21.242323 28150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.226979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:39975 (local address 127.4.28.1:38469)
0805 22:51:21.233775 (+ 6796us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:21.233797 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:21.233952 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:21.234393 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:21.234398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:21.234425 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:51:21.234821 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:21.234834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.236256 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.236261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:21.238641 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:21.238657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.240629 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.240635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:21.240807 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.240845 (+ 38us) client_negotiation.cc:769] Sending connection context
0805 22:51:21.241696 (+ 851us) client_negotiation.cc:240] Negotiation successful
0805 22:51:21.241868 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6647,"mutex_wait_us":1,"thread_start_us":10863,"threads_started":1}
I20260805 22:51:21.242676 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.227753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:50381 (local address 127.4.28.3:37095)
0805 22:51:21.231172 (+ 3419us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:21.231179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:21.234599 (+ 3420us) server_negotiation.cc:407] Connection header received
0805 22:51:21.234927 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:21.234933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:21.235023 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:21.235083 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:51:21.235871 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.236955 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.239301 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.240517 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.240622 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.240754 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.242212 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.242360 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3304,"thread_start_us":5246,"threads_started":1}
I20260805 22:51:21.243409 28148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.227167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:37095 (local address 127.4.28.1:50381)
0805 22:51:21.234484 (+ 7317us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:21.234513 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0805 22:51:21.234757 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:51:21.235258 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:51:21.235266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:51:21.235295 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:51:21.235686 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:21.235700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.237134 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.237139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:21.239124 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:51:21.239136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.241819 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.241826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:51:21.242011 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.242047 (+ 36us) client_negotiation.cc:769] Sending connection context
0805 22:51:21.242860 (+ 813us) client_negotiation.cc:240] Negotiation successful
0805 22:51:21.243007 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7168,"mutex_wait_us":38,"thread_start_us":6194,"threads_started":1}
I20260805 22:51:21.243721 28142 leader_election.cc:288] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [CANDIDATE]: Term 1 election: Requested vote from peers 21d417c90743455aa270caa2dfccba13 (127.4.28.1:35985), c74d9fc31fc24d70ab696c37509470b1 (127.4.28.3:37095)
I20260805 22:51:21.243876 27683 raft_consensus.cc:2454] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fff5363cf9449dbfe7cc060b19165a in term 1.
I20260805 22:51:21.243688 27858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fe6166c9114186997627744aa7f2f2" candidate_uuid: "21d417c90743455aa270caa2dfccba13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fff5363cf9449dbfe7cc060b19165a" is_pre_election: true
I20260805 22:51:21.244221 27858 raft_consensus.cc:2379] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 21d417c90743455aa270caa2dfccba13 in current term 1: Already voted for candidate 80fff5363cf9449dbfe7cc060b19165a in this term.
I20260805 22:51:21.244200 28044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fe6166c9114186997627744aa7f2f2" candidate_uuid: "21d417c90743455aa270caa2dfccba13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74d9fc31fc24d70ab696c37509470b1" is_pre_election: true
I20260805 22:51:21.244774 28045 raft_consensus.cc:2454] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fff5363cf9449dbfe7cc060b19165a in term 1.
I20260805 22:51:21.245325 27772 leader_election.cc:302] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21d417c90743455aa270caa2dfccba13, 80fff5363cf9449dbfe7cc060b19165a, c74d9fc31fc24d70ab696c37509470b1; no voters:
I20260805 22:51:21.245461 27585 leader_election.cc:302] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21d417c90743455aa270caa2dfccba13; no voters: 80fff5363cf9449dbfe7cc060b19165a, c74d9fc31fc24d70ab696c37509470b1
I20260805 22:51:21.245718 28142 raft_consensus.cc:2781] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:51:21.245952 28141 raft_consensus.cc:2726] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:51:21.246254 28142 raft_consensus.cc:686] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 1 LEADER]: Becoming Leader. State: Replica: 80fff5363cf9449dbfe7cc060b19165a, State: Running, Role: LEADER
I20260805 22:51:21.246788 28142 consensus_queue.cc:237] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } }
I20260805 22:51:21.251590 27472 catalog_manager.cc:5495] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fff5363cf9449dbfe7cc060b19165a (127.4.28.2). New cstate: current_term: 1 leader_uuid: "80fff5363cf9449dbfe7cc060b19165a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fff5363cf9449dbfe7cc060b19165a" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 39975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 } health_report { overall_health: UNKNOWN } } }
I20260805 22:51:21.301784 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.296002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.4.28.2:39975)
0805 22:51:21.296207 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:21.296214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:21.296354 (+ 140us) server_negotiation.cc:407] Connection header received
0805 22:51:21.296595 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:21.296600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:21.296696 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:21.296841 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:51:21.297266 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.298064 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.299478 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.299898 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.300010 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.300576 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:51:21.301112 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:51:21.301259 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.301373 (+ 114us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.301467 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 22:51:21.311343 28045 raft_consensus.cc:1261] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [term 1 FOLLOWER]: Refusing update from remote peer 80fff5363cf9449dbfe7cc060b19165a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:51:21.311534 27683 raft_consensus.cc:1261] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 1 FOLLOWER]: Refusing update from remote peer 80fff5363cf9449dbfe7cc060b19165a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:51:21.312353 28152 consensus_queue.cc:1035] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74d9fc31fc24d70ab696c37509470b1" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 37095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:51:21.312788 28142 consensus_queue.cc:1035] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d417c90743455aa270caa2dfccba13" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:51:21.326436 28158 mvcc.cc:205] Tried to move back new op lower bound from 7315334272239685632 to 7315334271990710272. Current Snapshot: MvccSnapshot[applied={T|T < 7315334272239685632}]
I20260805 22:51:21.342700 27472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:51:21.343118 27472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:51:21.347424 27472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Sending DeleteTablet for 3 replicas of tablet 21fe6166c9114186997627744aa7f2f2
I20260805 22:51:21.348812 27645 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fe6166c9114186997627744aa7f2f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:51:21 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:38880
I20260805 22:51:21.348840 27828 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fe6166c9114186997627744aa7f2f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:51:21 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:54474
I20260805 22:51:21.350055 28012 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fe6166c9114186997627744aa7f2f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:51:21 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:48416
I20260805 22:51:21.351091 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 27534
I20260805 22:51:21.359867 28180 tablet_replica.cc:330] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: stopping tablet replica
I20260805 22:51:21.360205 28179 tablet_replica.cc:330] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: stopping tablet replica
I20260805 22:51:21.360450 28180 raft_consensus.cc:2229] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:51:21.360617 28179 raft_consensus.cc:2229] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:51:21.361074 28179 raft_consensus.cc:2258] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:51:21.361068 28180 raft_consensus.cc:2258] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:51:21.362058 28177 tablet_replica.cc:330] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: stopping tablet replica
I20260805 22:51:21.362561 28177 raft_consensus.cc:2229] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:51:21.363469 28177 raft_consensus.cc:2258] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:51:21.363883 28180 ts_tablet_manager.cc:1905] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:51:21.366020 28179 ts_tablet_manager.cc:1905] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:51:21.371279 28177 ts_tablet_manager.cc:1905] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:51:21.371688 28180 ts_tablet_manager.cc:1918] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:51:21.371932 28180 log.cc:1196] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/21fe6166c9114186997627744aa7f2f2
I20260805 22:51:21.372689 28180 ts_tablet_manager.cc:1939] T 21fe6166c9114186997627744aa7f2f2 P c74d9fc31fc24d70ab696c37509470b1: Deleting consensus metadata
I20260805 22:51:21.374039 28179 ts_tablet_manager.cc:1918] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:51:21.374275 28179 log.cc:1196] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/21fe6166c9114186997627744aa7f2f2
I20260805 22:51:21.374867 28179 ts_tablet_manager.cc:1939] T 21fe6166c9114186997627744aa7f2f2 P 21d417c90743455aa270caa2dfccba13: Deleting consensus metadata
I20260805 22:51:21.375002 27453 catalog_manager.cc:4841] TS c74d9fc31fc24d70ab696c37509470b1 (127.4.28.3:37095): tablet 21fe6166c9114186997627744aa7f2f2 (table test-table [id=25377c25546d4baca6b79c49a8020b77]) successfully deleted
I20260805 22:51:21.376691 27457 catalog_manager.cc:4841] TS 21d417c90743455aa270caa2dfccba13 (127.4.28.1:35985): tablet 21fe6166c9114186997627744aa7f2f2 (table test-table [id=25377c25546d4baca6b79c49a8020b77]) successfully deleted
I20260805 22:51:21.378125 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.352955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.4.28.1:35985)
0805 22:51:21.353183 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:21.353189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:21.353210 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:51:21.353273 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:21.353279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:21.353361 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:21.353464 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:21.355701 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.356932 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.359032 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.359752 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.359907 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.365322 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:21.365342 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:21.365363 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:21.365430 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:21.373127 (+ 7697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:21.374527 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:21.374540 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:21.374544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:21.374746 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:21.375333 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:21.375339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:21.375342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:21.376486 (+ 1144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:21.376820 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.377508 (+ 688us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.377786 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:51:21.378743 27735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:51:21.384425 28177 ts_tablet_manager.cc:1918] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:51:21.384740 28177 log.cc:1196] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/21fe6166c9114186997627744aa7f2f2
I20260805 22:51:21.385490 28177 ts_tablet_manager.cc:1939] T 21fe6166c9114186997627744aa7f2f2 P 80fff5363cf9449dbfe7cc060b19165a: Deleting consensus metadata
I20260805 22:51:21.387859 27454 catalog_manager.cc:4841] TS 80fff5363cf9449dbfe7cc060b19165a (127.4.28.2:39975): tablet 21fe6166c9114186997627744aa7f2f2 (table test-table [id=25377c25546d4baca6b79c49a8020b77]) successfully deleted
I20260805 22:51:21.473841 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 27534
I20260805 22:51:21.507102 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 27747
I20260805 22:51:21.526306 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.507981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.4.28.2:39975)
0805 22:51:21.508218 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:21.508223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:21.508244 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:51:21.508423 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:21.508427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:21.508543 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:21.508633 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:21.510111 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.510872 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.512230 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.512767 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.512891 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.519771 (+ 6880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:21.519791 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:21.519810 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:21.519860 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:21.523164 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:21.523999 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:21.524009 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:21.524012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:21.524177 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:21.524592 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:21.524597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:21.524599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:21.525064 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:21.525362 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.525726 (+ 364us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.525976 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260805 22:51:21.527156 27920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:51:21.645054 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 27747
I20260805 22:51:21.672799 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 27931
I20260805 22:51:21.687208 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:21.673672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.4.28.3:37095)
0805 22:51:21.673883 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:21.673888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:21.673910 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:51:21.674109 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:21.674114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:21.674198 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:21.674290 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:21.675700 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.676415 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.677783 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:21.678234 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:21.678360 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:21.681041 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:21.681061 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:21.681080 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:21.681126 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:21.684010 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:21.684841 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:21.684851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:21.684853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:21.685000 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:21.685444 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:21.685449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:21.685451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:21.685954 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:21.686274 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:21.686604 (+ 330us) server_negotiation.cc:299] Negotiation successful
0805 22:51:21.686852 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260805 22:51:21.687867 28107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:51:21.778098 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 27931
I20260805 22:51:21.803264 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 27430
I20260805 22:51:21.804557 27526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:51:21.923746 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 27430
I20260805 22:51:41.953088 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34627
--webserver_interface=127.4.28.62
--webserver_port=41965
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:34627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:51:42.318414 28186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:51:42.318809 28186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:51:42.318950 28186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:51:42.326645 28186 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260805 22:51:42.326781 28186 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260805 22:51:42.326849 28186 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260805 22:51:42.326926 28186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:51:42.327008 28186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:51:42.327109 28186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:51:42.327178 28186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:51:42.327253 28186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:51:42.337865 28186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:34627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:51:42.339304 28186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:51:42.340803 28186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:51:42.353823 28200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:42.353798 28193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:42.353909 28197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:51:42.354063 28186 server_base.cc:1034] running on GCE node
I20260805 22:51:42.355295 28186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:51:42.356964 28186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:51:42.358187 28186 hybrid_clock.cc:648] HybridClock initialized: now 1785970302358117 us; error 46 us; skew 500 ppm
I20260805 22:51:42.358538 28186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:51:42 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970302, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:51:42.361904 28186 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:51:42.363391 28186 webserver.cc:460] Webserver started at http://127.4.28.62:41965/ using document root <none> and password file <none>
I20260805 22:51:42.364136 28186 fs_manager.cc:359] Metadata directory not provided
I20260805 22:51:42.364234 28186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:51:42.370426 28186 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260805 22:51:42.373816 28214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:42.375350 28186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 22:51:42.375552 28186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f"
format_stamp: "Formatted at 2026-08-05 22:50:54 on dist-test-slave-7lbf"
server_key: "bf8dab32a3208b2f9e3f1046e3deff4a"
server_key_iv: "b36c7465511365d2d119de166568bcba"
server_key_version: "encryptionkey@0"
I20260805 22:51:42.376610 28186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:51:42.401005 28186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:51:42.405789 28186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:51:42.406086 28186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:51:42.424983 28186 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:34627
I20260805 22:51:42.425043 28285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:34627 every 8 connection(s)
I20260805 22:51:42.426608 28186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:51:42.428933 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 28186
I20260805 22:51:42.429994 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:35985
--local_ip_for_outbound_sockets=127.4.28.1
--tserver_master_addrs=127.4.28.62:34627
--webserver_port=42595
--webserver_interface=127.4.28.1
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:51:42.436488 28286 sys_catalog.cc:263] Verifying existing consensus state
I20260805 22:51:42.441345 28286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Bootstrap starting.
I20260805 22:51:42.464136 28286 log.cc:824] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Log is configured to *not* fsync() on all Append() calls
I20260805 22:51:42.485209 28286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:51:42.485878 28286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Bootstrap complete.
I20260805 22:51:42.495667 28286 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:51:42.496743 28286 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fdc5ff3fcd48378a29acfdc0f2710f, State: Initialized, Role: FOLLOWER
I20260805 22:51:42.497478 28286 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:51:42.497776 28286 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:51:42.497972 28286 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:51:42.498165 28286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:51:42.502607 28286 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:51:42.503185 28286 leader_election.cc:302] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1fdc5ff3fcd48378a29acfdc0f2710f; no voters:
I20260805 22:51:42.503906 28286 leader_election.cc:288] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260805 22:51:42.504056 28292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:51:42.505163 28292 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [term 3 LEADER]: Becoming Leader. State: Replica: a1fdc5ff3fcd48378a29acfdc0f2710f, State: Running, Role: LEADER
I20260805 22:51:42.506289 28286 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:51:42.506215 28292 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } }
I20260805 22:51:42.509702 28293 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } } }
I20260805 22:51:42.510061 28293 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: This master's current role is: LEADER
I20260805 22:51:42.509310 28294 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fdc5ff3fcd48378a29acfdc0f2710f. Latest consensus state: current_term: 3 leader_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fdc5ff3fcd48378a29acfdc0f2710f" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 34627 } } }
I20260805 22:51:42.510787 28294 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f [sys.catalog]: This master's current role is: LEADER
I20260805 22:51:42.521731 28303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:51:42.533278 28303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:51:42.535161 28303 catalog_manager.cc:1259] Loaded cluster ID: 729745043b6342dcba34024a453b8f3e
I20260805 22:51:42.535390 28303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:51:42.539875 28303 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:51:42.542006 28303 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Loaded TSK: 0
I20260805 22:51:42.542310 28303 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1fdc5ff3fcd48378a29acfdc0f2710f: Loaded TSK: 1
F20260805 22:51:42.543877 28303 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f54b499e9ad google::LogMessage::Fail() at ??:0
@ 0x7f54b49a188a google::LogMessage::SendToLog() at ??:0
@ 0x7f54b499e4de google::LogMessage::Flush() at ??:0
@ 0x7f54b499fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f54b87027d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f54b86f173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f54ca3cd1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f54ca3cc783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f54ca4582a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f54ca4580c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f54ca4a25e9 std::function<>::operator()() at ??:0
@ 0x7f54ca3d0cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f54ca3cec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f54ca456610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f54ca456439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f54b75ea345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f54b75ed91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f54b75ed799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f54b75c8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f54ba8a26db start_thread at ??:0
@ 0x7f54b383861f clone at ??:0
W20260805 22:51:42.869249 28290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:51:42.869642 28290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:51:42.869797 28290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:51:42.877453 28290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:51:42.877642 28290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:51:42.877807 28290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:51:42.888239 28290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:35985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=42595
--tserver_master_addrs=127.4.28.62:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:51:42.889781 28290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:51:42.891271 28290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:51:42.905805 28351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:42.905720 28352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:42.905736 28354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:51:42.906997 28290 server_base.cc:1034] running on GCE node
I20260805 22:51:42.907666 28290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:51:42.908758 28290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:51:42.909967 28290 hybrid_clock.cc:648] HybridClock initialized: now 1785970302909905 us; error 67 us; skew 500 ppm
I20260805 22:51:42.910301 28290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:51:42 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970302, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:51:42.913662 28290 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:51:42.915266 28290 webserver.cc:460] Webserver started at http://127.4.28.1:42595/ using document root <none> and password file <none>
I20260805 22:51:42.915977 28290 fs_manager.cc:359] Metadata directory not provided
I20260805 22:51:42.916085 28290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:51:42.922137 28290 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:51:42.925237 28368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:42.926822 28290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:51:42.927022 28290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "21d417c90743455aa270caa2dfccba13"
format_stamp: "Formatted at 2026-08-05 22:50:55 on dist-test-slave-7lbf"
server_key: "8848b9239fe5bbf32360cbbef7697da6"
server_key_iv: "8c596d9d3e988ff94e652c4578fb5522"
server_key_version: "encryptionkey@0"
I20260805 22:51:42.928066 28290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:51:42.951597 28290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:51:42.956437 28290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:51:42.956708 28290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:51:42.957919 28290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:51:42.959501 28290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:51:42.959635 28290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:42.959748 28290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:51:42.959810 28290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:42.994484 28290 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:35985
I20260805 22:51:42.994566 28530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:35985 every 8 connection(s)
I20260805 22:51:42.996109 28290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:51:43.001132 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 28290
I20260805 22:51:43.002015 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:39975
--local_ip_for_outbound_sockets=127.4.28.2
--tserver_master_addrs=127.4.28.62:34627
--webserver_port=41449
--webserver_interface=127.4.28.2
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:51:43.002468 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:42.999649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:41329)
0805 22:51:43.000669 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:43.000991 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":374,"threads_started":1}
W20260805 22:51:43.003315 28531 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:34627 (0 consecutive failures): Network error: Failed to ping master at 127.4.28.62:34627: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
I20260805 22:51:43.004434 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:43.004001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:42877)
0805 22:51:43.004195 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:43.004288 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260805 22:51:43.005429 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:43.005045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:32863)
0805 22:51:43.005195 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:43.005286 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260805 22:51:43.005709 28531 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 22:51:43.365384 28534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:51:43.365736 28534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:51:43.365863 28534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:51:43.373500 28534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:51:43.373659 28534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:51:43.373813 28534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:51:43.384115 28534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:39975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=41449
--tserver_master_addrs=127.4.28.62:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:51:43.385632 28534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:51:43.387127 28534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:51:43.402757 28541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:43.402866 28542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:43.402824 28544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:51:43.404130 28534 server_base.cc:1034] running on GCE node
I20260805 22:51:43.404822 28534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:51:43.405990 28534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:51:43.407240 28534 hybrid_clock.cc:648] HybridClock initialized: now 1785970303407179 us; error 39 us; skew 500 ppm
I20260805 22:51:43.407590 28534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:51:43 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:51:43.413573 28534 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:51:43.415335 28534 webserver.cc:460] Webserver started at http://127.4.28.2:41449/ using document root <none> and password file <none>
I20260805 22:51:43.416111 28534 fs_manager.cc:359] Metadata directory not provided
I20260805 22:51:43.416246 28534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:51:43.422406 28534 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260805 22:51:43.425740 28556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:43.427297 28534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:51:43.427477 28534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "80fff5363cf9449dbfe7cc060b19165a"
format_stamp: "Formatted at 2026-08-05 22:50:56 on dist-test-slave-7lbf"
server_key: "670640efc9bb08f70efc62ca17e175b9"
server_key_iv: "c8880aa68abe617736e2384929f1c4da"
server_key_version: "encryptionkey@0"
I20260805 22:51:43.428571 28534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:51:43.449811 28534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:51:43.454155 28534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:51:43.454433 28534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:51:43.455528 28534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:51:43.457070 28534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:51:43.457185 28534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:43.457309 28534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:51:43.457381 28534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:43.490564 28534 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:39975
I20260805 22:51:43.490630 28712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:39975 every 8 connection(s)
I20260805 22:51:43.492180 28534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:51:43.497352 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 28534
I20260805 22:51:43.497742 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:43.495432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:46903)
0805 22:51:43.496240 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:43.496455 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":318,"threads_started":1}
I20260805 22:51:43.498162 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:37095
--local_ip_for_outbound_sockets=127.4.28.3
--tserver_master_addrs=127.4.28.62:34627
--webserver_port=45157
--webserver_interface=127.4.28.3
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:51:43.498629 28713 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:34627 (0 consecutive failures): Network error: Failed to ping master at 127.4.28.62:34627: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
I20260805 22:51:43.499673 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:43.499236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:50329)
0805 22:51:43.499430 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:43.499522 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260805 22:51:43.500707 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:43.500324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:39203)
0805 22:51:43.500474 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:43.500569 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260805 22:51:43.500972 28713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 22:51:43.857292 28716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:51:43.857642 28716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:51:43.857788 28716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:51:43.865258 28716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:51:43.865398 28716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:51:43.865571 28716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:51:43.875898 28716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:37095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=45157
--tserver_master_addrs=127.4.28.62:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:51:43.877277 28716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:51:43.878691 28716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:51:43.892342 28724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:43.892421 28725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:51:43.892369 28727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:51:43.893087 28716 server_base.cc:1034] running on GCE node
I20260805 22:51:43.893834 28716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:51:43.894868 28716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:51:43.896113 28716 hybrid_clock.cc:648] HybridClock initialized: now 1785970303896055 us; error 52 us; skew 500 ppm
I20260805 22:51:43.896449 28716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:51:43 dist-test-slave-7lbf krb5kdc[26557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:51:43.899789 28716 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:51:43.901585 28716 webserver.cc:460] Webserver started at http://127.4.28.3:45157/ using document root <none> and password file <none>
I20260805 22:51:43.902369 28716 fs_manager.cc:359] Metadata directory not provided
I20260805 22:51:43.902488 28716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:51:43.908425 28716 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 22:51:43.911624 28737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:43.913121 28716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:51:43.913291 28716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "c74d9fc31fc24d70ab696c37509470b1"
format_stamp: "Formatted at 2026-08-05 22:50:57 on dist-test-slave-7lbf"
server_key: "a6b5928e0b7acb5a250357ae9ac11be7"
server_key_iv: "4dbe66b65414a8a5858099a3fcf3b012"
server_key_version: "encryptionkey@0"
I20260805 22:51:43.914413 28716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:51:43.935117 28716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:51:43.939754 28716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:51:43.940017 28716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:51:43.941093 28716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:51:43.942605 28716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:51:43.942706 28716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:43.942793 28716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:51:43.942854 28716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:51:43.977404 28716 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:37095
I20260805 22:51:43.977454 28909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:37095 every 8 connection(s)
I20260805 22:51:43.978960 28716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:51:43.982195 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 28716
I20260805 22:51:43.984653 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:43.982201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:49009)
0805 22:51:43.983157 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:43.983433 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":355,"threads_started":1}
W20260805 22:51:43.985387 28910 heartbeater.cc:644] Failed to heartbeat to 127.4.28.62:34627 (0 consecutive failures): Network error: Failed to ping master at 127.4.28.62:34627: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
I20260805 22:51:43.986285 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:43.985928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:55949)
0805 22:51:43.986086 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:43.986171 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260805 22:51:43.987114 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:43.986791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:38003)
0805 22:51:43.986919 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:43.987000 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260805 22:51:43.987339 28910 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260805 22:51:44.007265 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:44.006373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:45987)
0805 22:51:44.006913 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:44.007019 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":270,"threads_started":1}
I20260805 22:51:44.502775 28916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:44.501821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:40241)
0805 22:51:44.502382 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:44.502484 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":274,"threads_started":1}
I20260805 22:51:44.989217 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:44.988077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:37695)
0805 22:51:44.988691 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:44.988858 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":268,"threads_started":1}
I20260805 22:51:45.009198 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:45.008289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:47537)
0805 22:51:45.008878 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:45.008977 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":270,"threads_started":1}
I20260805 22:51:45.506147 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:45.505307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:49647)
0805 22:51:45.505870 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:45.505958 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":296,"threads_started":1}
I20260805 22:51:45.991487 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:45.990514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:48601)
0805 22:51:45.991196 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:45.991305 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":407,"threads_started":1}
I20260805 22:51:46.011022 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:46.010231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:54735)
0805 22:51:46.010744 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:46.010836 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":277,"threads_started":1}
I20260805 22:51:46.508078 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:46.507172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:48765)
0805 22:51:46.507760 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:46.507853 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":324,"threads_started":1}
I20260805 22:51:46.993419 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:46.992554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:58963)
0805 22:51:46.993106 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:46.993224 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":272,"threads_started":1}
I20260805 22:51:47.012835 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:47.012034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:42645)
0805 22:51:47.012565 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:47.012651 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":271,"threads_started":1}
I20260805 22:51:47.510033 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:47.509196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:34387)
0805 22:51:47.509756 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:47.509845 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":247,"threads_started":1}
I20260805 22:51:47.995332 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:47.994501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:43009)
0805 22:51:47.995041 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:47.995132 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":275,"threads_started":1}
I20260805 22:51:48.014802 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:48.013924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:59991)
0805 22:51:48.014523 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:48.014618 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":275,"threads_started":1}
I20260805 22:51:48.512082 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:48.511210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:49121)
0805 22:51:48.511776 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:48.511897 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":299,"threads_started":1}
I20260805 22:51:48.997231 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:48.996375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:56897)
0805 22:51:48.996960 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:48.997051 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":278,"threads_started":1}
I20260805 22:51:49.016666 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:49.015862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:54997)
0805 22:51:49.016367 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:49.016495 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
I20260805 22:51:49.514036 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:49.513143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:41343)
0805 22:51:49.513723 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:49.513825 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":284,"threads_started":1}
I20260805 22:51:49.999212 28944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:49.998329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:42747)
0805 22:51:49.998926 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:49.999041 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":309,"threads_started":1}
I20260805 22:51:50.018800 28945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:50.017869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:59723)
0805 22:51:50.018475 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:50.018573 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":240,"threads_started":1}
I20260805 22:51:50.515962 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:50.515082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:34073)
0805 22:51:50.515646 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:50.515733 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":245,"threads_started":1}
I20260805 22:51:51.001365 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:51.000384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:34337)
0805 22:51:51.001026 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:51.001111 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":292,"threads_started":1}
I20260805 22:51:51.020752 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:51.019917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:34263)
0805 22:51:51.020471 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:51.020568 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":271,"threads_started":1}
I20260805 22:51:51.517802 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:51.517014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:57105)
0805 22:51:51.517492 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:51.517622 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":237,"threads_started":1}
I20260805 22:51:52.003219 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:52.002445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:43985)
0805 22:51:52.002955 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:52.003049 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":255,"threads_started":1}
I20260805 22:51:52.022627 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:52.021811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:33787)
0805 22:51:52.022341 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:52.022440 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":231,"threads_started":1}
I20260805 22:51:52.519793 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:52.518880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:55697)
0805 22:51:52.519503 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:52.519603 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":336,"threads_started":1}
I20260805 22:51:53.005352 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:53.004357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:33303)
0805 22:51:53.005020 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:53.005134 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":317,"threads_started":1}
I20260805 22:51:53.024658 28959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:53.023735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:32997)
0805 22:51:53.024337 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:53.024461 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":301,"threads_started":1}
I20260805 22:51:53.521831 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:53.520999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:41667)
0805 22:51:53.521557 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:53.521656 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":279,"threads_started":1}
I20260805 22:51:54.007632 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:54.006607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:56195)
0805 22:51:54.007231 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:54.007363 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":380,"threads_started":1}
I20260805 22:51:54.026785 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:54.025860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:39173)
0805 22:51:54.026445 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:54.026569 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":277,"threads_started":1}
I20260805 22:51:54.523598 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:54.522793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:60815)
0805 22:51:54.523312 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:54.523408 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":269,"threads_started":1}
I20260805 22:51:55.009711 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:55.008784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:41523)
0805 22:51:55.009336 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:55.009439 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260805 22:51:55.028677 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:55.027884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:40947)
0805 22:51:55.028399 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:55.028514 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":273,"threads_started":1}
I20260805 22:51:55.525503 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:55.524650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:38849)
0805 22:51:55.525223 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:55.525325 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":290,"threads_started":1}
I20260805 22:51:56.011525 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:56.010675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:51709)
0805 22:51:56.011246 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:56.011345 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
I20260805 22:51:56.030726 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:56.029794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:36987)
0805 22:51:56.030378 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:56.030494 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":281,"threads_started":1}
I20260805 22:51:56.527526 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:56.526659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:44623)
0805 22:51:56.527211 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:56.527309 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":266,"threads_started":1}
I20260805 22:51:57.013600 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:57.012608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:39969)
0805 22:51:57.013225 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:57.013321 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":304,"threads_started":1}
I20260805 22:51:57.032632 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:57.031786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:47647)
0805 22:51:57.032337 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:57.032437 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":310,"threads_started":1}
I20260805 22:51:57.529328 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:57.528549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:35023)
0805 22:51:57.529039 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:57.529126 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":236,"threads_started":1}
I20260805 22:51:58.015440 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:58.014618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:59435)
0805 22:51:58.015143 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:58.015240 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
I20260805 22:51:58.034678 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:58.033854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:60539)
0805 22:51:58.034370 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:58.034459 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":266,"threads_started":1}
I20260805 22:51:58.531453 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:58.530658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.2:43179)
0805 22:51:58.531161 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:58.531250 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":229,"threads_started":1}
I20260805 22:51:58.990202 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 28290
I20260805 22:51:59.015684 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:58.991651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.4.28.1:35985)
0805 22:51:58.992318 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:58.992341 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:58.992397 (+ 56us) server_negotiation.cc:407] Connection header received
0805 22:51:58.994320 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:58.994333 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:58.994817 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:58.995224 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:58.996946 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:58.998833 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:59.000696 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:59.001707 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:59.001871 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:59.008897 (+ 7026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:59.008908 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:59.008929 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:59.008978 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:59.012116 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:59.012876 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:59.012883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:59.012891 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:59.013106 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:59.013609 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:59.013613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:59.013615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:59.014225 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:59.014474 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:59.014912 (+ 438us) server_negotiation.cc:299] Negotiation successful
0805 22:51:59.015380 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":285,"threads_started":1}
I20260805 22:51:59.017196 28523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:51:59.017544 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:59.016663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.3:33571)
0805 22:51:59.017201 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:59.017286 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":318,"threads_started":1}
I20260805 22:51:59.104081 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:59.103024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:34627 (local address 127.4.28.1:33853)
0805 22:51:59.103627 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0805 22:51:59.103792 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.62:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":285,"threads_started":1}
I20260805 22:51:59.104058 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 28290
I20260805 22:51:59.133634 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 28534
I20260805 22:51:59.153375 28981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:59.134778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.4.28.2:39975)
0805 22:51:59.135307 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:59.135322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:59.135364 (+ 42us) server_negotiation.cc:407] Connection header received
0805 22:51:59.136917 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:59.136928 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:59.137393 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:59.137789 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:59.139340 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:59.140729 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:59.142246 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:59.143024 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:59.143192 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:59.145973 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:59.145983 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:59.146002 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:59.146053 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:59.149253 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:59.150230 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:59.150240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:59.150248 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:59.150490 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:59.151034 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:59.151039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:59.151041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:59.151691 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:59.151972 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:59.152619 (+ 647us) server_negotiation.cc:299] Negotiation successful
0805 22:51:59.153039 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
I20260805 22:51:59.155002 28706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:51:59.241791 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 28534
I20260805 22:51:59.267236 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 28716
I20260805 22:51:59.287077 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:51:59.268317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.4.28.3:37095)
0805 22:51:59.268884 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0805 22:51:59.268898 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:51:59.268942 (+ 44us) server_negotiation.cc:407] Connection header received
0805 22:51:59.270696 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:51:59.270708 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:51:59.271176 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:51:59.271543 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:51:59.273193 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:59.274776 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:59.276178 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:51:59.276942 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:51:59.277154 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:51:59.279997 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:51:59.280006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:51:59.280026 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:51:59.280075 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:51:59.283421 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:59.284215 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:59.284223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:59.284232 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:59.284431 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:51:59.284857 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:51:59.284861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:51:59.284863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:51:59.285456 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:51:59.285805 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0805 22:51:59.286272 (+ 467us) server_negotiation.cc:299] Negotiation successful
0805 22:51:59.286757 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":239,"threads_started":1}
I20260805 22:51:59.288612 28901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:51:59.374614 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 28716
2026-08-05T22:51:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66467 ms)
[----------] 30 tests from SecurityITest (316886 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:51:59 dist-test-slave-7lbf krb5kdc[28988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:51:59 dist-test-slave-7lbf krb5kdc[28988](info): set up 2 sockets
Aug 05 22:51:59 dist-test-slave-7lbf krb5kdc[28988](info): commencing operation
krb5kdc: starting...
W20260805 22:52:00.623461 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.002s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:00 dist-test-slave-7lbf krb5kdc[28988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:52:00.980932 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:39051
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:39051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:01.356671 29003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:01.357033 29003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:01.357177 29003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:01.364789 29003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:01.364951 29003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:01.365031 29003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:01.365113 29003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:01.365176 29003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:01.376055 29003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:39051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:01.377640 29003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:01.379180 29003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:01.393509 29010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:01.393797 29003 server_base.cc:1034] running on GCE node
W20260805 22:52:01.393460 29014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:01.393615 29011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:01.395059 29003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:01.396862 29003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:01.398154 29003 hybrid_clock.cc:648] HybridClock initialized: now 1785970321398082 us; error 62 us; skew 500 ppm
I20260805 22:52:01.398584 29003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:01 dist-test-slave-7lbf krb5kdc[28988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970321, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:01.402125 29003 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:52:01.403993 29003 webserver.cc:460] Webserver started at http://127.4.28.62:36263/ using document root <none> and password file <none>
I20260805 22:52:01.404816 29003 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:01.404940 29003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:01.405278 29003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:01.408944 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "cdc20d029efd4eeab18435dc7c411471"
format_stamp: "Formatted at 2026-08-05 22:52:01 on dist-test-slave-7lbf"
server_key: "23b9f887f6b71f1f57fbc5631457863f"
server_key_iv: "faecae173d144abbeb945c8ed3c96c54"
server_key_version: "encryptionkey@0"
I20260805 22:52:01.409924 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "cdc20d029efd4eeab18435dc7c411471"
format_stamp: "Formatted at 2026-08-05 22:52:01 on dist-test-slave-7lbf"
server_key: "23b9f887f6b71f1f57fbc5631457863f"
server_key_iv: "faecae173d144abbeb945c8ed3c96c54"
server_key_version: "encryptionkey@0"
I20260805 22:52:01.416173 29003 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260805 22:52:01.420346 29029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:01.421969 29003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260805 22:52:01.422185 29003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "cdc20d029efd4eeab18435dc7c411471"
format_stamp: "Formatted at 2026-08-05 22:52:01 on dist-test-slave-7lbf"
server_key: "23b9f887f6b71f1f57fbc5631457863f"
server_key_iv: "faecae173d144abbeb945c8ed3c96c54"
server_key_version: "encryptionkey@0"
I20260805 22:52:01.422429 29003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:01.460103 29003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:01.464768 29003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:01.465102 29003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:01.484509 29003 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:39051
I20260805 22:52:01.484529 29105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:39051 every 8 connection(s)
I20260805 22:52:01.486122 29003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:01.489650 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 29003
I20260805 22:52:01.489866 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:01.490454 29106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:01.490615 4208 external_mini_cluster.cc:1428] Setting key 0993d2addc9d35357dd1ef493e7dac15
I20260805 22:52:01.507189 29106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471: Bootstrap starting.
Aug 05 22:52:01 dist-test-slave-7lbf krb5kdc[28988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:01.511381 29106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:01.512718 29106 log.cc:824] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:01.516141 29106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471: No bootstrap required, opened a new log
I20260805 22:52:01.518194 29109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:01.492905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.4.28.62:39051)
0805 22:52:01.493995 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:01.494012 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:01.494077 (+ 65us) server_negotiation.cc:407] Connection header received
0805 22:52:01.496083 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:01.496125 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:01.496829 (+ 704us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:01.497424 (+ 595us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:01.499080 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:01.501105 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:01.502936 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:01.504221 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:01.504628 (+ 407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:01.509772 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:01.509784 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:01.509802 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:01.509854 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:01.513341 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:01.514259 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:01.514267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:01.514276 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:01.514482 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:01.514943 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:01.514947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:01.514950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:01.515562 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:01.515885 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:01.516437 (+ 552us) server_negotiation.cc:299] Negotiation successful
0805 22:52:01.516957 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":417,"threads_started":1}
I20260805 22:52:01.524749 29106 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc20d029efd4eeab18435dc7c411471" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39051 } }
I20260805 22:52:01.525240 29106 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:01.525347 29106 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc20d029efd4eeab18435dc7c411471, State: Initialized, Role: FOLLOWER
I20260805 22:52:01.526098 29106 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc20d029efd4eeab18435dc7c411471" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39051 } }
I20260805 22:52:01.526414 29106 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:01.526628 29106 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:01.526852 29106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:01.529392 29106 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc20d029efd4eeab18435dc7c411471" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39051 } }
I20260805 22:52:01.530040 29106 leader_election.cc:302] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc20d029efd4eeab18435dc7c411471; no voters:
I20260805 22:52:01.530802 29106 leader_election.cc:288] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:01.531049 29112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:01.532023 29112 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [term 1 LEADER]: Becoming Leader. State: Replica: cdc20d029efd4eeab18435dc7c411471, State: Running, Role: LEADER
I20260805 22:52:01.532621 29112 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc20d029efd4eeab18435dc7c411471" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39051 } }
I20260805 22:52:01.533330 29106 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:01.535936 29115 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc20d029efd4eeab18435dc7c411471. Latest consensus state: current_term: 1 leader_uuid: "cdc20d029efd4eeab18435dc7c411471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc20d029efd4eeab18435dc7c411471" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39051 } } }
I20260805 22:52:01.535961 29114 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc20d029efd4eeab18435dc7c411471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc20d029efd4eeab18435dc7c411471" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39051 } } }
I20260805 22:52:01.536482 29115 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:01.536597 29114 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:01.538919 29122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:01.545301 29122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:01.558135 29122 catalog_manager.cc:1347] Generated new cluster ID: c893024a096b4aa5a195a0f1c531bad9
I20260805 22:52:01.558355 29122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:01.571022 29122 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:01.573146 29122 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:01.590056 29122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471: Generated new TSK 0
I20260805 22:52:01.591403 29122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:52:01.685721 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39051
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:02.072221 29143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:02.072592 29143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:02.072739 29143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:02.080561 29143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:02.080718 29143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:02.080883 29143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:52:02.091555 29143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:02.093140 29143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:02.094700 29143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:02.109722 29150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:02.109687 29151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:02.109754 29155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:02.110613 29143 server_base.cc:1034] running on GCE node
I20260805 22:52:02.111316 29143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:02.112533 29143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:02.113770 29143 hybrid_clock.cc:648] HybridClock initialized: now 1785970322113690 us; error 73 us; skew 500 ppm
I20260805 22:52:02.114173 29143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:02 dist-test-slave-7lbf krb5kdc[28988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:02.118486 29143 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:52:02.120405 29143 webserver.cc:460] Webserver started at http://127.4.28.1:35677/ using document root <none> and password file <none>
I20260805 22:52:02.121220 29143 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:02.121361 29143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:02.121774 29143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:02.125628 29143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e"
format_stamp: "Formatted at 2026-08-05 22:52:02 on dist-test-slave-7lbf"
server_key: "c9594779355173fb9c0ea69a96d1d069"
server_key_iv: "d80d2e4de18dec0684fd26bb750cbfb6"
server_key_version: "encryptionkey@0"
I20260805 22:52:02.126744 29143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e"
format_stamp: "Formatted at 2026-08-05 22:52:02 on dist-test-slave-7lbf"
server_key: "c9594779355173fb9c0ea69a96d1d069"
server_key_iv: "d80d2e4de18dec0684fd26bb750cbfb6"
server_key_version: "encryptionkey@0"
I20260805 22:52:02.133116 29143 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260805 22:52:02.137455 29162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:02.139122 29143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:52:02.139338 29143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e"
format_stamp: "Formatted at 2026-08-05 22:52:02 on dist-test-slave-7lbf"
server_key: "c9594779355173fb9c0ea69a96d1d069"
server_key_iv: "d80d2e4de18dec0684fd26bb750cbfb6"
server_key_version: "encryptionkey@0"
I20260805 22:52:02.139598 29143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:02.175377 29143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:02.180286 29143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:02.180673 29143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:02.181840 29143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:02.183414 29143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:02.183497 29143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:02.183633 29143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:02.183691 29143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:02.217545 29143 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:43025
I20260805 22:52:02.217669 29333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:43025 every 8 connection(s)
I20260805 22:52:02.219206 29143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:52:02.224511 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 29143
I20260805 22:52:02.224718 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:52:02.225416 4208 external_mini_cluster.cc:1428] Setting key e3736d531f7b59d1b6248cb0bcfbfa43
Aug 05 22:52:02 dist-test-slave-7lbf krb5kdc[28988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:02.243278 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:02.222201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:40209 (local address 127.4.28.62:39051)
0805 22:52:02.222699 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:02.222704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:02.224000 (+ 1296us) server_negotiation.cc:407] Connection header received
0805 22:52:02.226277 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:02.226283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:02.226372 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:02.226489 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:02.228423 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:02.229441 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:02.231234 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:02.231791 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:02.231926 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:02.236168 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:02.236175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:02.236179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:02.236222 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:02.239173 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:02.240279 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:02.240284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:02.240287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:02.240455 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:02.240985 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:02.240988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:02.240991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:02.241282 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:02.241413 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:02.242523 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0805 22:52:02.242799 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":286,"threads_started":1}
I20260805 22:52:02.244174 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:02.222894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:39051 (local address 127.4.28.1:40209)
0805 22:52:02.223779 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:02.223832 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:02.225861 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:02.226810 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:02.226820 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:02.227563 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:02.228223 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:02.228243 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:02.229646 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:02.229651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:02.231070 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:02.231085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:02.232074 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:02.232080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:02.232390 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:02.233168 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:52:02.233209 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:02.235822 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:02.239371 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:02.239382 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:02.239403 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:02.240120 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:02.240596 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:02.240600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:02.240603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:02.240874 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:02.241408 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:02.241416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:02.241957 (+ 541us) client_negotiation.cc:769] Sending connection context
0805 22:52:02.242313 (+ 356us) client_negotiation.cc:240] Negotiation successful
0805 22:52:02.242804 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":365,"threads_started":1}
I20260805 22:52:02.245973 29334 heartbeater.cc:347] Connected to a master server at 127.4.28.62:39051
I20260805 22:52:02.246321 29334 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:02.247139 29334 heartbeater.cc:511] Master 127.4.28.62:39051 requested a full tablet report, sending...
I20260805 22:52:02.249804 29053 ts_manager.cc:194] Registered new tserver with Master: 0e4eda3a6ab54e57a29c7c6cd91f0e2e (127.4.28.1:43025)
I20260805 22:52:02.251380 29053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:40209
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:52:02.301898 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39051
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:02.673066 29344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:02.673367 29344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:02.673508 29344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:02.681181 29344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:02.681319 29344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:02.681475 29344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:52:02.692013 29344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:02.693492 29344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:02.695048 29344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:02.709628 29351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:02.709658 29350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:02.710245 29353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:02.711098 29344 server_base.cc:1034] running on GCE node
I20260805 22:52:02.711874 29344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:02.712996 29344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:02.714229 29344 hybrid_clock.cc:648] HybridClock initialized: now 1785970322714159 us; error 60 us; skew 500 ppm
I20260805 22:52:02.714581 29344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:02 dist-test-slave-7lbf krb5kdc[28988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:02.717944 29344 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:52:02.719604 29344 webserver.cc:460] Webserver started at http://127.4.28.2:36359/ using document root <none> and password file <none>
I20260805 22:52:02.720458 29344 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:02.720613 29344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:02.720938 29344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:02.724298 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "d55a7eeabd4643f78950925f2ad72024"
format_stamp: "Formatted at 2026-08-05 22:52:02 on dist-test-slave-7lbf"
server_key: "449f59b18b4e846b444e38d76817608a"
server_key_iv: "bdc7d374667d3c065fdd3f09e93a33c3"
server_key_version: "encryptionkey@0"
I20260805 22:52:02.725211 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "d55a7eeabd4643f78950925f2ad72024"
format_stamp: "Formatted at 2026-08-05 22:52:02 on dist-test-slave-7lbf"
server_key: "449f59b18b4e846b444e38d76817608a"
server_key_iv: "bdc7d374667d3c065fdd3f09e93a33c3"
server_key_version: "encryptionkey@0"
I20260805 22:52:02.731175 29344 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260805 22:52:02.735334 29361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:02.736934 29344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 22:52:02.737133 29344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "d55a7eeabd4643f78950925f2ad72024"
format_stamp: "Formatted at 2026-08-05 22:52:02 on dist-test-slave-7lbf"
server_key: "449f59b18b4e846b444e38d76817608a"
server_key_iv: "bdc7d374667d3c065fdd3f09e93a33c3"
server_key_version: "encryptionkey@0"
I20260805 22:52:02.737349 29344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:02.757866 29344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:02.762825 29344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:02.763150 29344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:02.764254 29344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:02.765841 29344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:02.765929 29344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:02.766079 29344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:02.766136 29344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:02.804008 29344 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:46139
I20260805 22:52:02.804085 29531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:46139 every 8 connection(s)
I20260805 22:52:02.805435 29344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:52:02.808413 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 29344
I20260805 22:52:02.808636 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:52:02.809271 4208 external_mini_cluster.cc:1428] Setting key 6eb5739ba164ae416e6412fd423d4aa0
Aug 05 22:52:02 dist-test-slave-7lbf krb5kdc[28988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:02.832221 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:02.808502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:55827 (local address 127.4.28.62:39051)
0805 22:52:02.810982 (+ 2480us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:02.810988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:02.811021 (+ 33us) server_negotiation.cc:407] Connection header received
0805 22:52:02.813164 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:02.813169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:02.813252 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:02.813351 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:02.815877 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:02.816872 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:02.819297 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:02.819780 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:02.819892 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:02.824801 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:02.824809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:02.824814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:02.824861 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:02.828054 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:02.829033 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:02.829039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:02.829043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:02.829225 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:02.829886 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:02.829892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:02.829895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:02.830372 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:02.830583 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:02.831568 (+ 985us) server_negotiation.cc:299] Negotiation successful
0805 22:52:02.831833 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2367,"thread_start_us":2250,"threads_started":1}
I20260805 22:52:02.833346 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:02.809187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:39051 (local address 127.4.28.2:55827)
0805 22:52:02.810318 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:02.810405 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:02.812750 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:02.813728 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:02.813742 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:02.814626 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:02.815559 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:02.815590 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:02.817104 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:02.817109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:02.819079 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:02.819099 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:02.820074 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:02.820081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:02.820402 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:02.821292 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:52:02.821332 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:02.824432 (+ 3100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:02.828249 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:02.828258 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:02.828275 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:02.828855 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:02.829396 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:02.829400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:02.829403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:02.829704 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:02.830584 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:02.830591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:02.831054 (+ 463us) client_negotiation.cc:769] Sending connection context
0805 22:52:02.831400 (+ 346us) client_negotiation.cc:240] Negotiation successful
0805 22:52:02.831893 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":408,"threads_started":1}
I20260805 22:52:02.835249 29532 heartbeater.cc:347] Connected to a master server at 127.4.28.62:39051
I20260805 22:52:02.835695 29532 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:02.836705 29532 heartbeater.cc:511] Master 127.4.28.62:39051 requested a full tablet report, sending...
I20260805 22:52:02.838650 29053 ts_manager.cc:194] Registered new tserver with Master: d55a7eeabd4643f78950925f2ad72024 (127.4.28.2:46139)
I20260805 22:52:02.839582 29053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:55827
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:52:02.893273 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39051
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:52:03.254889 29334 heartbeater.cc:503] Master 127.4.28.62:39051 was elected leader, sending a full tablet report...
W20260805 22:52:03.263561 29539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:03.263880 29539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:03.264010 29539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:03.271591 29539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:03.271730 29539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:03.271880 29539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:52:03.282248 29539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:03.283664 29539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:03.285099 29539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:03.299404 29546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:03.299441 29550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:03.299628 29552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:03.300388 29539 server_base.cc:1034] running on GCE node
I20260805 22:52:03.301142 29539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:03.302294 29539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:03.303568 29539 hybrid_clock.cc:648] HybridClock initialized: now 1785970323303504 us; error 51 us; skew 500 ppm
I20260805 22:52:03.303962 29539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:03 dist-test-slave-7lbf krb5kdc[28988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970323, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:03.307680 29539 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:52:03.309504 29539 webserver.cc:460] Webserver started at http://127.4.28.3:42577/ using document root <none> and password file <none>
I20260805 22:52:03.310319 29539 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:03.310456 29539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:03.310822 29539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:03.314155 29539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "e59a559d2bb248a4bb0e915326cb8e88"
format_stamp: "Formatted at 2026-08-05 22:52:03 on dist-test-slave-7lbf"
server_key: "bb60a252eeec7d97f7907e3f999f3c5d"
server_key_iv: "556a4ce05f33dc7af1b2b17d2f9675d9"
server_key_version: "encryptionkey@0"
I20260805 22:52:03.315058 29539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "e59a559d2bb248a4bb0e915326cb8e88"
format_stamp: "Formatted at 2026-08-05 22:52:03 on dist-test-slave-7lbf"
server_key: "bb60a252eeec7d97f7907e3f999f3c5d"
server_key_iv: "556a4ce05f33dc7af1b2b17d2f9675d9"
server_key_version: "encryptionkey@0"
I20260805 22:52:03.320863 29539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260805 22:52:03.325083 29559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:03.326922 29539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:52:03.327136 29539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "e59a559d2bb248a4bb0e915326cb8e88"
format_stamp: "Formatted at 2026-08-05 22:52:03 on dist-test-slave-7lbf"
server_key: "bb60a252eeec7d97f7907e3f999f3c5d"
server_key_iv: "556a4ce05f33dc7af1b2b17d2f9675d9"
server_key_version: "encryptionkey@0"
I20260805 22:52:03.327404 29539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:03.362462 29539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:03.367373 29539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:03.367651 29539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:03.368732 29539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:03.370430 29539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:03.370517 29539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:03.370682 29539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:03.370738 29539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:03.404487 29539 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:41411
I20260805 22:52:03.404521 29719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:41411 every 8 connection(s)
I20260805 22:52:03.405926 29539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:52:03.410411 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 29539
I20260805 22:52:03.410607 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:52:03.411063 4208 external_mini_cluster.cc:1428] Setting key 914a8878c4c657bdddba5415b3b51677
Aug 05 22:52:03 dist-test-slave-7lbf krb5kdc[28988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970323, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:03.426424 29722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.408588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:33877 (local address 127.4.28.62:39051)
0805 22:52:03.409053 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:03.409058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:03.410025 (+ 967us) server_negotiation.cc:407] Connection header received
0805 22:52:03.412103 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:03.412109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:03.412201 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:03.412296 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:03.413982 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.414765 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.416381 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.416779 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.416902 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.420560 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:03.420566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:03.420569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:03.420605 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:03.423146 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:03.423951 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:03.423955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:03.423958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:03.424094 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:03.424541 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:03.424545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:03.424547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:03.424835 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:03.424975 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:03.425765 (+ 790us) server_negotiation.cc:299] Negotiation successful
0805 22:52:03.426021 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":233,"threads_started":1}
I20260805 22:52:03.427577 29724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.409025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:39051 (local address 127.4.28.3:33877)
0805 22:52:03.409827 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:03.409887 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:03.411821 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:03.412577 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:03.412587 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:03.413140 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:03.413806 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.413830 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.414935 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.414939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.416234 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.416247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.417031 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.417040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.417280 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.417979 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:52:03.418010 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:03.420307 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:03.423307 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:03.423317 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:03.423336 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:03.423813 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:03.424207 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:03.424212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:03.424216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:03.424419 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:03.424968 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:03.424976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:03.425340 (+ 364us) client_negotiation.cc:769] Sending connection context
0805 22:52:03.425645 (+ 305us) client_negotiation.cc:240] Negotiation successful
0805 22:52:03.426284 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":336,"threads_started":1}
I20260805 22:52:03.428969 29720 heartbeater.cc:347] Connected to a master server at 127.4.28.62:39051
I20260805 22:52:03.429292 29720 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:03.430105 29720 heartbeater.cc:511] Master 127.4.28.62:39051 requested a full tablet report, sending...
I20260805 22:52:03.431686 29053 ts_manager.cc:194] Registered new tserver with Master: e59a559d2bb248a4bb0e915326cb8e88 (127.4.28.3:41411)
I20260805 22:52:03.432619 29053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:33877
I20260805 22:52:03.439258 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:52:03.458704 29722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.444329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.4.28.62:39051)
0805 22:52:03.444574 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:03.444580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:03.444941 (+ 361us) server_negotiation.cc:407] Connection header received
0805 22:52:03.445200 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:03.445205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:03.445298 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:03.445383 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:03.446989 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.447733 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.449106 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.449553 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.449670 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.452294 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:03.452299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:03.452303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:03.452344 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:03.455210 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:03.456254 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:03.456261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:03.456263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:03.456435 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:03.456890 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:03.456895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:03.456897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:03.457303 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:03.457597 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:03.458037 (+ 440us) server_negotiation.cc:299] Negotiation successful
0805 22:52:03.458329 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260805 22:52:03.463294 29053 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:52:03.470441 29053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:52:03.514942 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.484452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.0.0.1:37740)
0805 22:52:03.500259 (+ 15807us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:03.500288 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:03.500481 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:03.504199 (+ 3718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:03.504209 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:03.504246 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:03.504785 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.504801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.507331 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.507338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.510182 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.510208 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.513794 (+ 3586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.513801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.514059 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.514117 (+ 58us) client_negotiation.cc:769] Sending connection context
0805 22:52:03.514261 (+ 144us) client_negotiation.cc:240] Negotiation successful
0805 22:52:03.514475 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15642,"thread_start_us":10992,"threads_started":1}
I20260805 22:52:03.515826 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.484874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.4.28.1:43025)
0805 22:52:03.495040 (+ 10166us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:03.495053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:03.501117 (+ 6064us) server_negotiation.cc:407] Connection header received
0805 22:52:03.501268 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:03.501279 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:03.502208 (+ 929us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:03.502465 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:03.505050 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.507089 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.511437 (+ 4348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.513172 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.513319 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.513623 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:03.515281 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0805 22:52:03.515493 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9907,"thread_start_us":10181,"threads_started":1}
I20260805 22:52:03.520422 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.483817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:46139 (local address 127.0.0.1:48966)
0805 22:52:03.495909 (+ 12092us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:03.495977 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:03.496186 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:03.511783 (+ 15597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:03.511789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:03.511824 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:03.512274 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.512289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.514575 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.514582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.517303 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.517323 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.519588 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.519595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.519790 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.519827 (+ 37us) client_negotiation.cc:769] Sending connection context
0805 22:52:03.519907 (+ 80us) client_negotiation.cc:240] Negotiation successful
0805 22:52:03.520059 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11811,"thread_start_us":20027,"threads_started":1}
I20260805 22:52:03.520650 29235 tablet_service.cc:1467] Processing CreateTablet for tablet eb4351ce8ac643748e0ee25e0f46df06 (DEFAULT_TABLE table=test-table [id=244248c3d9de492b8b44f6bc6bdc0dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:03.521727 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.484044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.4.28.2:46139)
0805 22:52:03.510971 (+ 26927us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:03.510983 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:03.511021 (+ 38us) server_negotiation.cc:407] Connection header received
0805 22:52:03.511145 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:03.511153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:03.511501 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:03.512551 (+ 1050us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:03.512802 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.514404 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.517748 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.519302 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.520889 (+ 1587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.521105 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:03.521180 (+ 75us) server_negotiation.cc:299] Negotiation successful
0805 22:52:03.521348 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26687,"thread_start_us":26514,"threads_started":1}
I20260805 22:52:03.522543 29235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4351ce8ac643748e0ee25e0f46df06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:03.522696 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.496452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.4.28.3:41411)
0805 22:52:03.511440 (+ 14988us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:03.511449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:03.511485 (+ 36us) server_negotiation.cc:407] Connection header received
0805 22:52:03.511585 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:03.511592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:03.511853 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:03.512061 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:03.513070 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.514669 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.517843 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.519410 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.519545 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.519776 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:03.522092 (+ 2316us) server_negotiation.cc:299] Negotiation successful
0805 22:52:03.522342 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14799,"thread_start_us":14622,"threads_started":1}
I20260805 22:52:03.523094 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.495712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:41411 (local address 127.0.0.1:35098)
0805 22:52:03.503621 (+ 7909us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:03.503646 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:03.503782 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:03.512380 (+ 8598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:03.512386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:03.512415 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:03.512840 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.512852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.514879 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.514885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.517614 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.517632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.519941 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.519947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.520132 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.520168 (+ 36us) client_negotiation.cc:769] Sending connection context
0805 22:52:03.520255 (+ 87us) client_negotiation.cc:240] Negotiation successful
0805 22:52:03.522510 (+ 2255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7729,"mutex_wait_us":105,"thread_start_us":4152,"threads_started":1}
I20260805 22:52:03.525877 29431 tablet_service.cc:1467] Processing CreateTablet for tablet eb4351ce8ac643748e0ee25e0f46df06 (DEFAULT_TABLE table=test-table [id=244248c3d9de492b8b44f6bc6bdc0dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:03.526259 29628 tablet_service.cc:1467] Processing CreateTablet for tablet eb4351ce8ac643748e0ee25e0f46df06 (DEFAULT_TABLE table=test-table [id=244248c3d9de492b8b44f6bc6bdc0dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:03.527515 29431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4351ce8ac643748e0ee25e0f46df06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:03.527642 29628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4351ce8ac643748e0ee25e0f46df06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:03.539381 29751 tablet_bootstrap.cc:492] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e: Bootstrap starting.
I20260805 22:52:03.541365 29752 tablet_bootstrap.cc:492] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: Bootstrap starting.
I20260805 22:52:03.543588 29753 tablet_bootstrap.cc:492] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: Bootstrap starting.
I20260805 22:52:03.544262 29751 tablet_bootstrap.cc:654] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:03.545105 29752 tablet_bootstrap.cc:654] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:03.545841 29751 log.cc:824] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:03.546207 29752 log.cc:824] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:03.547412 29753 tablet_bootstrap.cc:654] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:03.548609 29753 log.cc:824] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:03.549283 29751 tablet_bootstrap.cc:492] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e: No bootstrap required, opened a new log
I20260805 22:52:03.549471 29752 tablet_bootstrap.cc:492] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: No bootstrap required, opened a new log
I20260805 22:52:03.549646 29751 ts_tablet_manager.cc:1397] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260805 22:52:03.549893 29752 ts_tablet_manager.cc:1397] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260805 22:52:03.551977 29753 tablet_bootstrap.cc:492] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: No bootstrap required, opened a new log
I20260805 22:52:03.552253 29753 ts_tablet_manager.cc:1397] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260805 22:52:03.559542 29753 raft_consensus.cc:348] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } }
I20260805 22:52:03.559979 29753 raft_consensus.cc:374] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:03.560081 29753 raft_consensus.cc:729] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55a7eeabd4643f78950925f2ad72024, State: Initialized, Role: FOLLOWER
I20260805 22:52:03.560760 29753 consensus_queue.cc:260] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } }
I20260805 22:52:03.560915 29752 raft_consensus.cc:348] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } }
I20260805 22:52:03.560915 29751 raft_consensus.cc:348] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } }
I20260805 22:52:03.561417 29752 raft_consensus.cc:374] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:03.561417 29751 raft_consensus.cc:374] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:03.561550 29751 raft_consensus.cc:729] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4eda3a6ab54e57a29c7c6cd91f0e2e, State: Initialized, Role: FOLLOWER
I20260805 22:52:03.561555 29752 raft_consensus.cc:729] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59a559d2bb248a4bb0e915326cb8e88, State: Initialized, Role: FOLLOWER
I20260805 22:52:03.562259 29752 consensus_queue.cc:260] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } }
I20260805 22:52:03.562259 29751 consensus_queue.cc:260] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } }
I20260805 22:52:03.564399 29532 heartbeater.cc:503] Master 127.4.28.62:39051 was elected leader, sending a full tablet report...
I20260805 22:52:03.565232 29753 ts_tablet_manager.cc:1428] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260805 22:52:03.566743 29720 heartbeater.cc:503] Master 127.4.28.62:39051 was elected leader, sending a full tablet report...
I20260805 22:52:03.567013 29751 ts_tablet_manager.cc:1428] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20260805 22:52:03.567358 29752 ts_tablet_manager.cc:1428] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
W20260805 22:52:03.657745 29721 tablet.cc:2369] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:52:03.665472 29758 raft_consensus.cc:482] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:52:03.665875 29758 raft_consensus.cc:504] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } }
I20260805 22:52:03.667769 29758 leader_election.cc:288] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e4eda3a6ab54e57a29c7c6cd91f0e2e (127.4.28.1:43025), e59a559d2bb248a4bb0e915326cb8e88 (127.4.28.3:41411)
I20260805 22:52:03.677886 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.668339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:53669 (local address 127.4.28.3:41411)
0805 22:52:03.668555 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:03.668561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:03.669339 (+ 778us) server_negotiation.cc:407] Connection header received
0805 22:52:03.669645 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:03.669651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:03.669738 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:03.669848 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:03.670653 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.671783 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.675203 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.676520 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.676691 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.676812 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:03.677423 (+ 611us) server_negotiation.cc:299] Negotiation successful
0805 22:52:03.677606 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260805 22:52:03.677937 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.668341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:52011 (local address 127.4.28.1:43025)
0805 22:52:03.668570 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:03.668576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:03.669374 (+ 798us) server_negotiation.cc:407] Connection header received
0805 22:52:03.669645 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:03.669651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:03.669747 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:03.669848 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:03.670653 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.671814 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.675215 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.676545 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.676691 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.676812 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:03.677441 (+ 629us) server_negotiation.cc:299] Negotiation successful
0805 22:52:03.677612 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260805 22:52:03.678157 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.668204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:41411 (local address 127.4.28.2:53669)
0805 22:52:03.669188 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:03.669211 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:03.669404 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:03.669936 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:03.669940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:03.669958 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:03.670466 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.670476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.671978 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.671984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.674917 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.674945 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.676870 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.676878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.677177 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.677253 (+ 76us) client_negotiation.cc:769] Sending connection context
0805 22:52:03.677369 (+ 116us) client_negotiation.cc:240] Negotiation successful
0805 22:52:03.677631 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":416,"thread_start_us":534,"threads_started":1}
I20260805 22:52:03.678539 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.668204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.4.28.2:52011)
0805 22:52:03.669259 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:03.669272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:03.669420 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:03.669894 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:03.669899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:03.669930 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:03.670372 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.670411 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.671978 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.671984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.674917 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:03.674945 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.676872 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.676878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:03.677172 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.677253 (+ 81us) client_negotiation.cc:769] Sending connection context
0805 22:52:03.677369 (+ 116us) client_negotiation.cc:240] Negotiation successful
0805 22:52:03.677631 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":282,"spinlock_wait_cycles":49920,"thread_start_us":578,"threads_started":1}
I20260805 22:52:03.679401 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4351ce8ac643748e0ee25e0f46df06" candidate_uuid: "d55a7eeabd4643f78950925f2ad72024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59a559d2bb248a4bb0e915326cb8e88" is_pre_election: true
I20260805 22:52:03.679543 29267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4351ce8ac643748e0ee25e0f46df06" candidate_uuid: "d55a7eeabd4643f78950925f2ad72024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" is_pre_election: true
I20260805 22:52:03.679955 29657 raft_consensus.cc:2454] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55a7eeabd4643f78950925f2ad72024 in term 0.
I20260805 22:52:03.680065 29267 raft_consensus.cc:2454] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55a7eeabd4643f78950925f2ad72024 in term 0.
I20260805 22:52:03.680708 29370 leader_election.cc:302] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e4eda3a6ab54e57a29c7c6cd91f0e2e, d55a7eeabd4643f78950925f2ad72024, e59a559d2bb248a4bb0e915326cb8e88; no voters:
I20260805 22:52:03.681169 29758 raft_consensus.cc:2781] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:52:03.681334 29758 raft_consensus.cc:482] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:52:03.681437 29758 raft_consensus.cc:3037] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:03.683743 29758 raft_consensus.cc:504] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } }
I20260805 22:52:03.684401 29758 leader_election.cc:288] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [CANDIDATE]: Term 1 election: Requested vote from peers 0e4eda3a6ab54e57a29c7c6cd91f0e2e (127.4.28.1:43025), e59a559d2bb248a4bb0e915326cb8e88 (127.4.28.3:41411)
I20260805 22:52:03.685017 29267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4351ce8ac643748e0ee25e0f46df06" candidate_uuid: "d55a7eeabd4643f78950925f2ad72024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e"
I20260805 22:52:03.685021 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4351ce8ac643748e0ee25e0f46df06" candidate_uuid: "d55a7eeabd4643f78950925f2ad72024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59a559d2bb248a4bb0e915326cb8e88"
I20260805 22:52:03.685213 29267 raft_consensus.cc:3037] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:03.685228 29657 raft_consensus.cc:3037] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:03.687541 29267 raft_consensus.cc:2454] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55a7eeabd4643f78950925f2ad72024 in term 1.
I20260805 22:52:03.687556 29657 raft_consensus.cc:2454] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55a7eeabd4643f78950925f2ad72024 in term 1.
I20260805 22:52:03.688002 29370 leader_election.cc:302] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d55a7eeabd4643f78950925f2ad72024, e59a559d2bb248a4bb0e915326cb8e88; no voters:
I20260805 22:52:03.688313 29758 raft_consensus.cc:2781] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:03.688802 29758 raft_consensus.cc:686] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 1 LEADER]: Becoming Leader. State: Replica: d55a7eeabd4643f78950925f2ad72024, State: Running, Role: LEADER
I20260805 22:52:03.689296 29758 consensus_queue.cc:237] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } }
I20260805 22:52:03.694335 29052 catalog_manager.cc:5495] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 reported cstate change: term changed from 0 to 1, leader changed from <none> to d55a7eeabd4643f78950925f2ad72024 (127.4.28.2). New cstate: current_term: 1 leader_uuid: "d55a7eeabd4643f78950925f2ad72024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55a7eeabd4643f78950925f2ad72024" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 } health_report { overall_health: UNKNOWN } } }
W20260805 22:52:03.722072 29335 tablet.cc:2369] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:52:03.751334 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.745801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.4.28.2:46139)
0805 22:52:03.746016 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:03.746022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:03.746222 (+ 200us) server_negotiation.cc:407] Connection header received
0805 22:52:03.746409 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:03.746413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:03.746509 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:03.746645 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:52:03.747028 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.747772 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.749252 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.749720 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.749818 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.750318 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:52:03.750864 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:52:03.750992 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:03.751031 (+ 39us) server_negotiation.cc:299] Negotiation successful
0805 22:52:03.751104 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:52:03.760267 29267 raft_consensus.cc:1261] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [term 1 FOLLOWER]: Refusing update from remote peer d55a7eeabd4643f78950925f2ad72024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:03.760273 29657 raft_consensus.cc:1261] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [term 1 FOLLOWER]: Refusing update from remote peer d55a7eeabd4643f78950925f2ad72024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:03.761373 29764 consensus_queue.cc:1035] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59a559d2bb248a4bb0e915326cb8e88" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:03.761793 29758 consensus_queue.cc:1035] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4eda3a6ab54e57a29c7c6cd91f0e2e" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:03.778424 29771 mvcc.cc:205] Tried to move back new op lower bound from 7315334446111133696 to 7315334445835513856. Current Snapshot: MvccSnapshot[applied={T|T < 7315334446111133696}]
I20260805 22:52:03.779031 29770 mvcc.cc:205] Tried to move back new op lower bound from 7315334446111133696 to 7315334445835513856. Current Snapshot: MvccSnapshot[applied={T|T < 7315334446111133696}]
I20260805 22:52:03.792848 29052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:03.793188 29052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:03.796973 29052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdc20d029efd4eeab18435dc7c411471: Sending DeleteTablet for 3 replicas of tablet eb4351ce8ac643748e0ee25e0f46df06
I20260805 22:52:03.798326 29431 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4351ce8ac643748e0ee25e0f46df06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:03 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:48966
I20260805 22:52:03.798354 29235 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4351ce8ac643748e0ee25e0f46df06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:03 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:37740
I20260805 22:52:03.798617 29628 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4351ce8ac643748e0ee25e0f46df06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:03 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:35098
I20260805 22:52:03.800396 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 29143
I20260805 22:52:03.811688 29786 tablet_replica.cc:330] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: stopping tablet replica
I20260805 22:52:03.812232 29786 raft_consensus.cc:2229] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:03.812778 29786 raft_consensus.cc:2258] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:03.815169 29786 ts_tablet_manager.cc:1905] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:03.823056 29786 ts_tablet_manager.cc:1918] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:03.823272 29786 log.cc:1196] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/eb4351ce8ac643748e0ee25e0f46df06
I20260805 22:52:03.823868 29786 ts_tablet_manager.cc:1939] T eb4351ce8ac643748e0ee25e0f46df06 P e59a559d2bb248a4bb0e915326cb8e88: Deleting consensus metadata
I20260805 22:52:03.826442 29034 catalog_manager.cc:4841] TS e59a559d2bb248a4bb0e915326cb8e88 (127.4.28.3:41411): tablet eb4351ce8ac643748e0ee25e0f46df06 (table test-table [id=244248c3d9de492b8b44f6bc6bdc0dbc]) successfully deleted
I20260805 22:52:03.849881 29785 tablet_replica.cc:330] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: stopping tablet replica
I20260805 22:52:03.850567 29785 raft_consensus.cc:2229] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:52:03.851548 29785 raft_consensus.cc:2258] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:03.852599 29784 tablet_replica.cc:330] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e: stopping tablet replica
I20260805 22:52:03.854420 29785 ts_tablet_manager.cc:1905] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 05 22:52:03 dist-test-slave-7lbf krb5kdc[28988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:52:03.857602 29784 raft_consensus.cc:2229] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:03.858317 29784 raft_consensus.cc:2258] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:03.862704 29785 ts_tablet_manager.cc:1918] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:03.862918 29785 log.cc:1196] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/eb4351ce8ac643748e0ee25e0f46df06
I20260805 22:52:03.863585 29785 ts_tablet_manager.cc:1939] T eb4351ce8ac643748e0ee25e0f46df06 P d55a7eeabd4643f78950925f2ad72024: Deleting consensus metadata
I20260805 22:52:03.864475 29784 ts_tablet_manager.cc:1905] T eb4351ce8ac643748e0ee25e0f46df06 P 0e4eda3a6ab54e57a29c7c6cd91f0e2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:03.865741 29031 catalog_manager.cc:4841] TS d55a7eeabd4643f78950925f2ad72024 (127.4.28.2:46139): tablet eb4351ce8ac643748e0ee25e0f46df06 (table test-table [id=244248c3d9de492b8b44f6bc6bdc0dbc]) successfully deleted
I20260805 22:52:03.866804 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:03.802056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.4.28.1:43025)
0805 22:52:03.802256 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:03.802263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:03.835433 (+ 33170us) server_negotiation.cc:407] Connection header received
0805 22:52:03.835772 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:03.835779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:03.835922 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:03.836006 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:03.845153 (+ 9147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.846619 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.852034 (+ 5415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:03.852779 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:03.852934 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:03.858672 (+ 5738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:03.858693 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:03.858713 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:03.858771 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:03.862777 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:03.863934 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:03.863945 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:03.863949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:03.864175 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:03.864765 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:03.864771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:03.864774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:03.865275 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:03.865612 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:03.866283 (+ 671us) server_negotiation.cc:299] Negotiation successful
0805 22:52:03.866481 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 22:52:03.868263 29326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:04.023367 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 29143
I20260805 22:52:04.049898 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 29344
W20260805 22:52:04.049902 29032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.28.1:43025 (error 108)
I20260805 22:52:04.052563 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.051484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.0.0.1:37756)
0805 22:52:04.052254 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:04.052385 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":514,"threads_started":1}
W20260805 22:52:04.052848 29032 catalog_manager.cc:4584] TS 0e4eda3a6ab54e57a29c7c6cd91f0e2e (127.4.28.1:43025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb4351ce8ac643748e0ee25e0f46df06: Network error: Client connection negotiation failed: client connection to 127.4.28.1:43025: connect: Connection refused (error 111)
Aug 05 22:52:04 dist-test-slave-7lbf krb5kdc[28988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:52:04.079254 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.051422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.4.28.2:46139)
0805 22:52:04.051610 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:04.051617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:04.051737 (+ 120us) server_negotiation.cc:407] Connection header received
0805 22:52:04.051935 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:04.051940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:04.052031 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:04.052132 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:04.054048 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:04.054891 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:04.056875 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:04.057333 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:04.057455 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:04.072378 (+ 14923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:04.072396 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:04.072415 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:04.072465 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:04.075653 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:04.076761 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:04.076770 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:04.076775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:04.077057 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:04.077642 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:04.077646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:04.077649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:04.078072 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:04.078329 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:04.078700 (+ 371us) server_negotiation.cc:299] Negotiation successful
0805 22:52:04.078912 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 22:52:04.080137 29523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:04.106652 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.106141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.0.0.1:37760)
0805 22:52:04.106368 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:04.106481 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260805 22:52:04.107786 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.107394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.0.0.1:37766)
0805 22:52:04.107551 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:04.107629 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260805 22:52:04.168547 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.168015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.0.0.1:37768)
0805 22:52:04.168274 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:04.168391 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260805 22:52:04.169689 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.169275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.0.0.1:37776)
0805 22:52:04.169400 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:04.169476 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260805 22:52:04.198007 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 29344
I20260805 22:52:04.226675 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 29539
Aug 05 22:52:04 dist-test-slave-7lbf krb5kdc[28988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:52:04.244155 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.227541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.4.28.3:41411)
0805 22:52:04.228096 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:04.228103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:04.228122 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:52:04.228182 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:04.228185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:04.228251 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:04.228359 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:04.229836 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:04.230622 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:04.232120 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:04.232592 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:04.232718 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:04.237696 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:04.237715 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:04.237734 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:04.237780 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:04.240782 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:04.241724 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:04.241733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:04.241736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:04.241938 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:04.242365 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:04.242369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:04.242371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:04.242825 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:04.243139 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:04.243545 (+ 406us) server_negotiation.cc:299] Negotiation successful
0805 22:52:04.243787 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":265,"threads_started":1}
I20260805 22:52:04.244800 29715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:04.250385 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.249933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.0.0.1:37792)
0805 22:52:04.250147 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:04.250237 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260805 22:52:04.251500 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.251070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.0.0.1:37798)
0805 22:52:04.251291 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:04.251361 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260805 22:52:04.335057 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 29539
I20260805 22:52:04.361020 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 29003
I20260805 22:52:04.362315 29100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:04.476159 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:04.475608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:43025 (local address 127.0.0.1:37808)
0805 22:52:04.475841 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:04.475962 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260805 22:52:04.476387 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 29003
2026-08-05T22:52:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5107 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-05T22:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:04Z Disabled control of system clock
I20260805 22:52:04.584154 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:38441
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:38441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 22:52:04.955361 29802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:04.955709 29802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:04.955794 29802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:04.963459 29802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:04.963608 29802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:04.963647 29802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:04.963706 29802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:04.963750 29802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:04.973471 29802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:38441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:38441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:04.974862 29802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:04.976843 29802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:04.989480 29811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:04.989439 29813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:04.989447 29815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:04.990324 29802 server_base.cc:1034] running on GCE node
I20260805 22:52:04.991225 29802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:04.992825 29802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:04.994048 29802 hybrid_clock.cc:648] HybridClock initialized: now 1785970324993982 us; error 56 us; skew 500 ppm
I20260805 22:52:04.994422 29802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:04.996980 29802 webserver.cc:460] Webserver started at http://127.4.28.62:34417/ using document root <none> and password file <none>
I20260805 22:52:04.997830 29802 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:04.997952 29802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:04.998304 29802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:05.001886 29802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "9010c5f31edc45f295ff2a9f89927bcf"
format_stamp: "Formatted at 2026-08-05 22:52:04 on dist-test-slave-7lbf"
server_key: "4ca675e13d7f733688667b00119af595"
server_key_iv: "3dd4610e6c746568e16e83523fc949ab"
server_key_version: "encryptionkey@0"
I20260805 22:52:05.002914 29802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "9010c5f31edc45f295ff2a9f89927bcf"
format_stamp: "Formatted at 2026-08-05 22:52:04 on dist-test-slave-7lbf"
server_key: "4ca675e13d7f733688667b00119af595"
server_key_iv: "3dd4610e6c746568e16e83523fc949ab"
server_key_version: "encryptionkey@0"
I20260805 22:52:05.010236 29802 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260805 22:52:05.014871 29824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:05.016762 29802 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:52:05.016987 29802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "9010c5f31edc45f295ff2a9f89927bcf"
format_stamp: "Formatted at 2026-08-05 22:52:04 on dist-test-slave-7lbf"
server_key: "4ca675e13d7f733688667b00119af595"
server_key_iv: "3dd4610e6c746568e16e83523fc949ab"
server_key_version: "encryptionkey@0"
I20260805 22:52:05.017241 29802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:05.060189 29802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:05.061174 29802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:05.061499 29802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:05.080538 29802 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:38441
I20260805 22:52:05.080556 29897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:38441 every 8 connection(s)
I20260805 22:52:05.082050 29802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:05.086244 29901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:05.090022 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 29802
I20260805 22:52:05.090200 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:05.090694 4208 external_mini_cluster.cc:1428] Setting key 668c5fcb1755591ca24c512a3bb0dfbf
I20260805 22:52:05.100284 29901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf: Bootstrap starting.
I20260805 22:52:05.103971 29901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:05.105348 29901 log.cc:824] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:05.106179 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:05.093201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.4.28.62:38441)
0805 22:52:05.094678 (+ 1477us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:05.094691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:05.094739 (+ 48us) server_negotiation.cc:407] Connection header received
0805 22:52:05.096478 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:05.096505 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:05.097083 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:05.097569 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:05.098216 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:05.099556 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:05.101138 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:05.101924 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:05.102114 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:05.103354 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:05.103363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:05.103380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:05.103990 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:05.104094 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:05.104120 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:05.104248 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:05.104554 (+ 306us) server_negotiation.cc:299] Negotiation successful
0805 22:52:05.104933 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":681,"threads_started":1}
I20260805 22:52:05.108934 29901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf: No bootstrap required, opened a new log
I20260805 22:52:05.117043 29901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9010c5f31edc45f295ff2a9f89927bcf" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38441 } }
I20260805 22:52:05.117496 29901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:05.117640 29901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9010c5f31edc45f295ff2a9f89927bcf, State: Initialized, Role: FOLLOWER
I20260805 22:52:05.118289 29901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9010c5f31edc45f295ff2a9f89927bcf" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38441 } }
I20260805 22:52:05.118553 29901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:05.118693 29901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:05.118901 29901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:05.121498 29901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9010c5f31edc45f295ff2a9f89927bcf" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38441 } }
I20260805 22:52:05.122182 29901 leader_election.cc:302] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9010c5f31edc45f295ff2a9f89927bcf; no voters:
I20260805 22:52:05.122859 29901 leader_election.cc:288] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:05.123054 29908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:05.123821 29908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [term 1 LEADER]: Becoming Leader. State: Replica: 9010c5f31edc45f295ff2a9f89927bcf, State: Running, Role: LEADER
I20260805 22:52:05.124403 29908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9010c5f31edc45f295ff2a9f89927bcf" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38441 } }
I20260805 22:52:05.125285 29901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:05.127899 29909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9010c5f31edc45f295ff2a9f89927bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9010c5f31edc45f295ff2a9f89927bcf" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38441 } } }
I20260805 22:52:05.128407 29909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:05.128235 29910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9010c5f31edc45f295ff2a9f89927bcf. Latest consensus state: current_term: 1 leader_uuid: "9010c5f31edc45f295ff2a9f89927bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9010c5f31edc45f295ff2a9f89927bcf" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 38441 } } }
I20260805 22:52:05.128566 29910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:05.131004 29917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:05.136847 29917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:05.160548 29917 catalog_manager.cc:1347] Generated new cluster ID: 628f11e3c34f4b9ca3ab84d5a630f81f
I20260805 22:52:05.160715 29917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:05.189556 29917 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:05.191604 29917 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:05.200042 29917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf: Generated new TSK 0
I20260805 22:52:05.201184 29917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:05.210901 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:38441
--builtin_ntp_servers=127.4.28.20:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 22:52:05.584244 29930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:05.584587 29930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:05.584741 29930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:05.592357 29930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:05.592486 29930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:05.592608 29930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:52:05.602105 29930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:38441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:05.603401 29930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:05.605336 29930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:05.619733 29938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:05.619827 29942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:05.619767 29936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:05.620451 29930 server_base.cc:1034] running on GCE node
I20260805 22:52:05.621230 29930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:05.622377 29930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:05.623591 29930 hybrid_clock.cc:648] HybridClock initialized: now 1785970325623539 us; error 46 us; skew 500 ppm
I20260805 22:52:05.623924 29930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:05.626451 29930 webserver.cc:460] Webserver started at http://127.4.28.1:36123/ using document root <none> and password file <none>
I20260805 22:52:05.627259 29930 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:05.627358 29930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:05.627702 29930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:05.631014 29930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "975a7c62511e478cac31ffd4a01a1d20"
format_stamp: "Formatted at 2026-08-05 22:52:05 on dist-test-slave-7lbf"
server_key: "3562dfd6a23905d20a9fd6630183b891"
server_key_iv: "6947044c2c04181a0557768235498ef3"
server_key_version: "encryptionkey@0"
I20260805 22:52:05.631860 29930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "975a7c62511e478cac31ffd4a01a1d20"
format_stamp: "Formatted at 2026-08-05 22:52:05 on dist-test-slave-7lbf"
server_key: "3562dfd6a23905d20a9fd6630183b891"
server_key_iv: "6947044c2c04181a0557768235498ef3"
server_key_version: "encryptionkey@0"
I20260805 22:52:05.637355 29930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260805 22:52:05.641340 29953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:05.643076 29930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:52:05.643368 29930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "975a7c62511e478cac31ffd4a01a1d20"
format_stamp: "Formatted at 2026-08-05 22:52:05 on dist-test-slave-7lbf"
server_key: "3562dfd6a23905d20a9fd6630183b891"
server_key_iv: "6947044c2c04181a0557768235498ef3"
server_key_version: "encryptionkey@0"
I20260805 22:52:05.643631 29930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:05.666427 29930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:05.667380 29930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:05.667687 29930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:05.668788 29930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:05.670576 29930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:05.670712 29930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:05.670831 29930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:05.670908 29930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:05.703413 29930 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:33143
I20260805 22:52:05.703491 30121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:33143 every 8 connection(s)
I20260805 22:52:05.704825 29930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:52:05.706784 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 29930
I20260805 22:52:05.706980 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:52:05.707495 4208 external_mini_cluster.cc:1428] Setting key 1f48f5fc88132ff820b5fc492ba992bb
I20260805 22:52:05.710675 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:38441
--builtin_ntp_servers=127.4.28.20:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 22:52:05.718958 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:05.707888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:35451 (local address 127.4.28.62:38441)
0805 22:52:05.708421 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:05.708428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:05.709924 (+ 1496us) server_negotiation.cc:407] Connection header received
0805 22:52:05.711869 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:05.711876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:05.711977 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:05.712097 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:05.713266 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:05.714274 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:05.715951 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:05.716515 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:05.716652 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:05.718111 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:05.718118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:05.718122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:05.718149 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:05.718203 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:05.718219 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:05.718310 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:05.718503 (+ 193us) server_negotiation.cc:299] Negotiation successful
0805 22:52:05.718612 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":265,"threads_started":1}
I20260805 22:52:05.719978 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:05.708557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:38441 (local address 127.4.28.1:35451)
0805 22:52:05.709754 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:05.709809 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:05.711558 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:05.712301 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:05.712311 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:05.712412 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:05.713069 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:05.713090 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:05.714441 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:05.714445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:05.715782 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:05.715796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:05.716727 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:05.716731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:05.716955 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:05.717743 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:52:05.717746 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:05.717792 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:52:05.717888 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:05.718305 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:05.718311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:05.718317 (+ 6us) client_negotiation.cc:769] Sending connection context
0805 22:52:05.718466 (+ 149us) client_negotiation.cc:240] Negotiation successful
0805 22:52:05.718822 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":329,"threads_started":1}
I20260805 22:52:05.721781 30122 heartbeater.cc:347] Connected to a master server at 127.4.28.62:38441
I20260805 22:52:05.722115 30122 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:05.722985 30122 heartbeater.cc:511] Master 127.4.28.62:38441 requested a full tablet report, sending...
I20260805 22:52:05.725283 29846 ts_manager.cc:194] Registered new tserver with Master: 975a7c62511e478cac31ffd4a01a1d20 (127.4.28.1:33143)
I20260805 22:52:05.727370 29846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.1:35451
W20260805 22:52:06.075268 30130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:06.075615 30130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:06.075757 30130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:06.083536 30130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:06.083676 30130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:06.083834 30130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:52:06.093638 30130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:38441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:06.095031 30130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:06.097076 30130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:06.111467 30138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:06.111431 30139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:06.111398 30142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:06.112145 30130 server_base.cc:1034] running on GCE node
I20260805 22:52:06.112830 30130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:06.113909 30130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:06.115159 30130 hybrid_clock.cc:648] HybridClock initialized: now 1785970326115100 us; error 52 us; skew 500 ppm
I20260805 22:52:06.115509 30130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:06.118088 30130 webserver.cc:460] Webserver started at http://127.4.28.2:42751/ using document root <none> and password file <none>
I20260805 22:52:06.118829 30130 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:06.118949 30130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:06.119287 30130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:06.122536 30130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "45f6c08513234baeb5ce1582c7b66ed4"
format_stamp: "Formatted at 2026-08-05 22:52:06 on dist-test-slave-7lbf"
server_key: "d154f4aa270839d16e1b88ae3fb8f459"
server_key_iv: "307a020fb1c1637ffad95ce877a07338"
server_key_version: "encryptionkey@0"
I20260805 22:52:06.123459 30130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "45f6c08513234baeb5ce1582c7b66ed4"
format_stamp: "Formatted at 2026-08-05 22:52:06 on dist-test-slave-7lbf"
server_key: "d154f4aa270839d16e1b88ae3fb8f459"
server_key_iv: "307a020fb1c1637ffad95ce877a07338"
server_key_version: "encryptionkey@0"
I20260805 22:52:06.129416 30130 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260805 22:52:06.133633 30149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:06.135159 30130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260805 22:52:06.135344 30130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "45f6c08513234baeb5ce1582c7b66ed4"
format_stamp: "Formatted at 2026-08-05 22:52:06 on dist-test-slave-7lbf"
server_key: "d154f4aa270839d16e1b88ae3fb8f459"
server_key_iv: "307a020fb1c1637ffad95ce877a07338"
server_key_version: "encryptionkey@0"
I20260805 22:52:06.135582 30130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:06.149104 30130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:06.150134 30130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:06.150420 30130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:06.151512 30130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:06.153193 30130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:06.153296 30130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:06.153417 30130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:06.153491 30130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:06.185468 30130 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:41333
I20260805 22:52:06.185542 30309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:41333 every 8 connection(s)
I20260805 22:52:06.187047 30130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:52:06.188931 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 30130
I20260805 22:52:06.189124 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:52:06.189694 4208 external_mini_cluster.cc:1428] Setting key fb7ede800d2213fb4431a2841592de73
I20260805 22:52:06.193246 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:38441
--builtin_ntp_servers=127.4.28.20:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 22:52:06.202884 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.190070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:47283 (local address 127.4.28.62:38441)
0805 22:52:06.190302 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:06.190308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:06.192219 (+ 1911us) server_negotiation.cc:407] Connection header received
0805 22:52:06.194785 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:06.194791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:06.194870 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:06.194966 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:06.196517 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.197292 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.199619 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.200106 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.200265 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.202058 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:06.202064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:06.202068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:06.202091 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:06.202129 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:06.202142 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:06.202227 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:06.202503 (+ 276us) server_negotiation.cc:299] Negotiation successful
0805 22:52:06.202607 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:52:06.204399 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.190813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:38441 (local address 127.4.28.2:47283)
0805 22:52:06.192025 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:06.192089 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:06.194442 (+ 2353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:06.195282 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:06.195297 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:06.195435 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:06.196290 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.196316 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.197498 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.197504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.199416 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.199436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.200387 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.200395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.200708 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.201621 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:52:06.201625 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:06.201684 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:52:06.201799 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:06.202250 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:06.202263 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:06.202272 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 22:52:06.202485 (+ 213us) client_negotiation.cc:240] Negotiation successful
0805 22:52:06.202950 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":367,"threads_started":1}
I20260805 22:52:06.206090 30310 heartbeater.cc:347] Connected to a master server at 127.4.28.62:38441
I20260805 22:52:06.206465 30310 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:06.207468 30310 heartbeater.cc:511] Master 127.4.28.62:38441 requested a full tablet report, sending...
I20260805 22:52:06.209391 29846 ts_manager.cc:194] Registered new tserver with Master: 45f6c08513234baeb5ce1582c7b66ed4 (127.4.28.2:41333)
I20260805 22:52:06.210654 29846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.2:47283
W20260805 22:52:06.558032 30316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:06.558359 30316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:06.558449 30316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:06.565905 30316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:06.566035 30316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:06.566146 30316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:52:06.575811 30316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:38441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:06.577204 30316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:06.579128 30316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:06.592926 30326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:06.592967 30324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:06.593060 30323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:06.593675 30316 server_base.cc:1034] running on GCE node
I20260805 22:52:06.594358 30316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:06.595412 30316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:06.596663 30316 hybrid_clock.cc:648] HybridClock initialized: now 1785970326596607 us; error 48 us; skew 500 ppm
I20260805 22:52:06.597038 30316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:06.599538 30316 webserver.cc:460] Webserver started at http://127.4.28.3:45069/ using document root <none> and password file <none>
I20260805 22:52:06.600299 30316 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:06.600414 30316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:06.600765 30316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:06.603988 30316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "4f8973c9f518414c933ab2aba1c289cf"
format_stamp: "Formatted at 2026-08-05 22:52:06 on dist-test-slave-7lbf"
server_key: "e2a1105fe27defa87a4ba39cb119acc7"
server_key_iv: "404565eb65510f2a97366cc7be60c64c"
server_key_version: "encryptionkey@0"
I20260805 22:52:06.604864 30316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "4f8973c9f518414c933ab2aba1c289cf"
format_stamp: "Formatted at 2026-08-05 22:52:06 on dist-test-slave-7lbf"
server_key: "e2a1105fe27defa87a4ba39cb119acc7"
server_key_iv: "404565eb65510f2a97366cc7be60c64c"
server_key_version: "encryptionkey@0"
I20260805 22:52:06.610764 30316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260805 22:52:06.614614 30333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:06.616117 30316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:52:06.616292 30316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "4f8973c9f518414c933ab2aba1c289cf"
format_stamp: "Formatted at 2026-08-05 22:52:06 on dist-test-slave-7lbf"
server_key: "e2a1105fe27defa87a4ba39cb119acc7"
server_key_iv: "404565eb65510f2a97366cc7be60c64c"
server_key_version: "encryptionkey@0"
I20260805 22:52:06.616526 30316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:06.641572 30316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:06.642581 30316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:06.642904 30316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:06.644037 30316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:06.645782 30316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:06.645877 30316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:06.645963 30316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:06.646020 30316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:06.680572 30316 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:44655
I20260805 22:52:06.680701 30494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:44655 every 8 connection(s)
I20260805 22:52:06.682021 30316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:52:06.682456 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 30316
I20260805 22:52:06.682643 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:52:06.683257 4208 external_mini_cluster.cc:1428] Setting key c88b3a75c857c582506189b69b3386ed
I20260805 22:52:06.696072 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.685056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:54421 (local address 127.4.28.62:38441)
0805 22:52:06.685283 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:06.685289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:06.687076 (+ 1787us) server_negotiation.cc:407] Connection header received
0805 22:52:06.689035 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:06.689041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:06.689119 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:06.689209 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:06.690484 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.691287 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.693128 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.693610 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.693726 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.695253 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:06.695259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:06.695263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:06.695286 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:06.695322 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:06.695335 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:06.695406 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:06.695677 (+ 271us) server_negotiation.cc:299] Negotiation successful
0805 22:52:06.695792 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260805 22:52:06.697227 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.685723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:38441 (local address 127.4.28.3:54421)
0805 22:52:06.686912 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:06.686956 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:06.688706 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:06.689450 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:06.689459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:06.689637 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:06.690317 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.690338 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.691481 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.691487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.692915 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.692934 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.693856 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.693863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.694142 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.694842 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:52:06.694845 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:06.694903 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:52:06.695039 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:06.695421 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:06.695427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:06.695434 (+ 7us) client_negotiation.cc:769] Sending connection context
0805 22:52:06.695601 (+ 167us) client_negotiation.cc:240] Negotiation successful
0805 22:52:06.695998 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":378,"threads_started":1}
I20260805 22:52:06.698800 30495 heartbeater.cc:347] Connected to a master server at 127.4.28.62:38441
I20260805 22:52:06.699149 30495 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:06.700042 30495 heartbeater.cc:511] Master 127.4.28.62:38441 requested a full tablet report, sending...
I20260805 22:52:06.701649 29846 ts_manager.cc:194] Registered new tserver with Master: 4f8973c9f518414c933ab2aba1c289cf (127.4.28.3:44655)
I20260805 22:52:06.702523 29846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.3:54421
I20260805 22:52:06.712355 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:52:06.723376 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.716889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.4.28.62:38441)
0805 22:52:06.717101 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:06.717108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:06.717558 (+ 450us) server_negotiation.cc:407] Connection header received
0805 22:52:06.717751 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:06.717756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:06.717837 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:06.717925 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:06.718468 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.719177 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.720678 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.721226 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.721371 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.722613 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:06.722618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:06.722622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:06.722643 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:06.722678 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:06.722690 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:06.722768 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:06.722916 (+ 148us) server_negotiation.cc:299] Negotiation successful
0805 22:52:06.723027 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 22:52:06.730895 30122 heartbeater.cc:503] Master 127.4.28.62:38441 was elected leader, sending a full tablet report...
I20260805 22:52:06.727763 29846 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:52:06.735432 29846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:52:06.762848 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.749681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:41333 (local address 127.0.0.1:34256)
0805 22:52:06.750666 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:06.750730 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:06.750970 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:06.752269 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:06.752281 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:06.752336 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:06.752796 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.752812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.755149 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.755155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.758067 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.758100 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.761171 (+ 3071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.761178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.762038 (+ 860us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.762115 (+ 77us) client_negotiation.cc:769] Sending connection context
0805 22:52:06.762239 (+ 124us) client_negotiation.cc:240] Negotiation successful
0805 22:52:06.762441 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":4,"thread_start_us":486,"threads_started":1}
I20260805 22:52:06.762966 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.749722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.4.28.2:41333)
0805 22:52:06.750397 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:06.750408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:06.751205 (+ 797us) server_negotiation.cc:407] Connection header received
0805 22:52:06.751345 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:06.751355 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:06.751631 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:06.751867 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:06.753077 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.754930 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.759230 (+ 4300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.760933 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.761110 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.761425 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:06.762392 (+ 967us) server_negotiation.cc:299] Negotiation successful
0805 22:52:06.762596 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":1585,"threads_started":1}
I20260805 22:52:06.767307 30519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.750776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.0.0.1:38354)
0805 22:52:06.758858 (+ 8082us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:06.758881 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:06.759023 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:06.760444 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:06.760450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:06.760480 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:06.760786 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.760795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.762779 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.762785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.764892 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.764910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.766478 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.766486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.766693 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.766729 (+ 36us) client_negotiation.cc:769] Sending connection context
0805 22:52:06.766821 (+ 92us) client_negotiation.cc:240] Negotiation successful
0805 22:52:06.766972 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7918,"thread_start_us":7713,"threads_started":1}
I20260805 22:52:06.767438 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.751637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.4.28.1:33143)
0805 22:52:06.759307 (+ 7670us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:06.759320 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:06.759365 (+ 45us) server_negotiation.cc:407] Connection header received
0805 22:52:06.759495 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:06.759506 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:06.759850 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:06.760140 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:06.760973 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.762585 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.765130 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.766238 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.766380 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.766687 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:06.766903 (+ 216us) server_negotiation.cc:299] Negotiation successful
0805 22:52:06.767100 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7420,"thread_start_us":8146,"threads_started":1}
I20260805 22:52:06.768610 30222 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae9dbaa53784a58bce67b59cde89046 (DEFAULT_TABLE table=test-table [id=2394e6372192487b915cd6c9347b8d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:06.769488 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.750275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:44655 (local address 127.0.0.1:45716)
0805 22:52:06.751008 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:06.751026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:06.751141 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:06.761348 (+ 10207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:06.761354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:06.761387 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:06.761827 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.761840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.764371 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.764376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.766516 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.766535 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.768660 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.768668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.768863 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.768902 (+ 39us) client_negotiation.cc:769] Sending connection context
0805 22:52:06.768974 (+ 72us) client_negotiation.cc:240] Negotiation successful
0805 22:52:06.769111 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":30,"thread_start_us":1245,"threads_started":1}
I20260805 22:52:06.770422 30222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae9dbaa53784a58bce67b59cde89046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:06.770881 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.751384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.4.28.3:44655)
0805 22:52:06.760577 (+ 9193us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:06.760588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:06.760631 (+ 43us) server_negotiation.cc:407] Connection header received
0805 22:52:06.760752 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:06.760761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:06.761073 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:06.761493 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:06.762067 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.764183 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.766781 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.768414 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.769910 (+ 1496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.770207 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:06.770294 (+ 87us) server_negotiation.cc:299] Negotiation successful
0805 22:52:06.770476 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8996,"thread_start_us":8810,"threads_started":1}
I20260805 22:52:06.771435 30025 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae9dbaa53784a58bce67b59cde89046 (DEFAULT_TABLE table=test-table [id=2394e6372192487b915cd6c9347b8d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:06.772694 30025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae9dbaa53784a58bce67b59cde89046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:06.774835 30405 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae9dbaa53784a58bce67b59cde89046 (DEFAULT_TABLE table=test-table [id=2394e6372192487b915cd6c9347b8d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:06.776458 30405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae9dbaa53784a58bce67b59cde89046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:06.784937 30526 tablet_bootstrap.cc:492] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: Bootstrap starting.
I20260805 22:52:06.789258 30526 tablet_bootstrap.cc:654] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:06.790943 30526 log.cc:824] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:06.797331 30528 tablet_bootstrap.cc:492] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: Bootstrap starting.
I20260805 22:52:06.797344 30530 tablet_bootstrap.cc:492] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: Bootstrap starting.
I20260805 22:52:06.801414 30528 tablet_bootstrap.cc:654] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:06.801805 30530 tablet_bootstrap.cc:654] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:06.802655 30528 log.cc:824] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:06.803165 30530 log.cc:824] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:06.804649 30526 tablet_bootstrap.cc:492] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: No bootstrap required, opened a new log
I20260805 22:52:06.805043 30526 ts_tablet_manager.cc:1397] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20260805 22:52:06.806238 30528 tablet_bootstrap.cc:492] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: No bootstrap required, opened a new log
I20260805 22:52:06.806525 30528 ts_tablet_manager.cc:1397] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260805 22:52:06.813024 30530 tablet_bootstrap.cc:492] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: No bootstrap required, opened a new log
I20260805 22:52:06.813380 30530 ts_tablet_manager.cc:1397] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260805 22:52:06.814230 30528 raft_consensus.cc:348] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } }
I20260805 22:52:06.814664 30528 raft_consensus.cc:374] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:06.814770 30528 raft_consensus.cc:729] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f6c08513234baeb5ce1582c7b66ed4, State: Initialized, Role: FOLLOWER
I20260805 22:52:06.815421 30528 consensus_queue.cc:260] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } }
I20260805 22:52:06.815850 30526 raft_consensus.cc:348] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } }
I20260805 22:52:06.816310 30526 raft_consensus.cc:374] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:06.816740 30310 heartbeater.cc:503] Master 127.4.28.62:38441 was elected leader, sending a full tablet report...
I20260805 22:52:06.818024 30528 ts_tablet_manager.cc:1428] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260805 22:52:06.816425 30526 raft_consensus.cc:729] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975a7c62511e478cac31ffd4a01a1d20, State: Initialized, Role: FOLLOWER
I20260805 22:52:06.819170 30526 consensus_queue.cc:260] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } }
I20260805 22:52:06.821211 30526 ts_tablet_manager.cc:1428] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260805 22:52:06.824239 30530 raft_consensus.cc:348] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } }
I20260805 22:52:06.824721 30530 raft_consensus.cc:374] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:06.824838 30530 raft_consensus.cc:729] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8973c9f518414c933ab2aba1c289cf, State: Initialized, Role: FOLLOWER
I20260805 22:52:06.825438 30530 consensus_queue.cc:260] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } }
I20260805 22:52:06.826642 30495 heartbeater.cc:503] Master 127.4.28.62:38441 was elected leader, sending a full tablet report...
I20260805 22:52:06.827293 30530 ts_tablet_manager.cc:1428] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260805 22:52:06.879164 30533 raft_consensus.cc:482] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:52:06.879473 30533 raft_consensus.cc:504] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } }
I20260805 22:52:06.881253 30533 leader_election.cc:288] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8973c9f518414c933ab2aba1c289cf (127.4.28.3:44655), 975a7c62511e478cac31ffd4a01a1d20 (127.4.28.1:33143)
I20260805 22:52:06.888327 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.881790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:54567 (local address 127.4.28.3:44655)
0805 22:52:06.882075 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:06.882082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:06.882800 (+ 718us) server_negotiation.cc:407] Connection header received
0805 22:52:06.883004 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:06.883010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:06.883076 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:06.883151 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:06.883729 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.884463 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.886473 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.887318 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.887447 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.887565 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:06.887937 (+ 372us) server_negotiation.cc:299] Negotiation successful
0805 22:52:06.888073 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260805 22:52:06.888326 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.881986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:44379 (local address 127.4.28.1:33143)
0805 22:52:06.882283 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:06.882289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:06.882882 (+ 593us) server_negotiation.cc:407] Connection header received
0805 22:52:06.883080 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:06.883085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:06.883158 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:06.883237 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:06.883899 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.884624 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.886519 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.887317 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.887455 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.887558 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:06.887922 (+ 364us) server_negotiation.cc:299] Negotiation successful
0805 22:52:06.888060 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260805 22:52:06.888583 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.881555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:44655 (local address 127.4.28.2:54567)
0805 22:52:06.882658 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:06.882680 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:06.882862 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:06.883188 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:06.883194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:06.883219 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:06.883564 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.883573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.884612 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.884616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.886249 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.886264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.887548 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.887556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.887713 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.887795 (+ 82us) client_negotiation.cc:769] Sending connection context
0805 22:52:06.887874 (+ 79us) client_negotiation.cc:240] Negotiation successful
0805 22:52:06.888054 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":70,"thread_start_us":923,"threads_started":1}
I20260805 22:52:06.888712 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.881798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.4.28.2:44379)
0805 22:52:06.882780 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:06.882795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:06.882920 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:06.883289 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:06.883294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:06.883336 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:06.883733 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.883743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.884777 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.884781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.886299 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:06.886317 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.887585 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.887595 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:06.887736 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.887780 (+ 44us) client_negotiation.cc:769] Sending connection context
0805 22:52:06.887874 (+ 94us) client_negotiation.cc:240] Negotiation successful
0805 22:52:06.888053 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":27,"thread_start_us":420,"threads_started":1}
I20260805 22:52:06.889693 30438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae9dbaa53784a58bce67b59cde89046" candidate_uuid: "45f6c08513234baeb5ce1582c7b66ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8973c9f518414c933ab2aba1c289cf" is_pre_election: true
I20260805 22:52:06.889693 30054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae9dbaa53784a58bce67b59cde89046" candidate_uuid: "45f6c08513234baeb5ce1582c7b66ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975a7c62511e478cac31ffd4a01a1d20" is_pre_election: true
I20260805 22:52:06.890249 30054 raft_consensus.cc:2454] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f6c08513234baeb5ce1582c7b66ed4 in term 0.
I20260805 22:52:06.890249 30438 raft_consensus.cc:2454] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f6c08513234baeb5ce1582c7b66ed4 in term 0.
I20260805 22:52:06.890988 30150 leader_election.cc:302] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f6c08513234baeb5ce1582c7b66ed4, 975a7c62511e478cac31ffd4a01a1d20; no voters:
I20260805 22:52:06.891404 30533 raft_consensus.cc:2781] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:52:06.891575 30533 raft_consensus.cc:482] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:52:06.891685 30533 raft_consensus.cc:3037] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:06.894052 30533 raft_consensus.cc:504] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } }
I20260805 22:52:06.894675 30533 leader_election.cc:288] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8973c9f518414c933ab2aba1c289cf (127.4.28.3:44655), 975a7c62511e478cac31ffd4a01a1d20 (127.4.28.1:33143)
I20260805 22:52:06.895329 30054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae9dbaa53784a58bce67b59cde89046" candidate_uuid: "45f6c08513234baeb5ce1582c7b66ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975a7c62511e478cac31ffd4a01a1d20"
I20260805 22:52:06.895329 30438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae9dbaa53784a58bce67b59cde89046" candidate_uuid: "45f6c08513234baeb5ce1582c7b66ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8973c9f518414c933ab2aba1c289cf"
I20260805 22:52:06.895570 30438 raft_consensus.cc:3037] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:06.895579 30054 raft_consensus.cc:3037] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:06.898964 30054 raft_consensus.cc:2454] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f6c08513234baeb5ce1582c7b66ed4 in term 1.
I20260805 22:52:06.898964 30438 raft_consensus.cc:2454] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f6c08513234baeb5ce1582c7b66ed4 in term 1.
I20260805 22:52:06.899554 30150 leader_election.cc:302] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45f6c08513234baeb5ce1582c7b66ed4, 4f8973c9f518414c933ab2aba1c289cf, 975a7c62511e478cac31ffd4a01a1d20; no voters:
I20260805 22:52:06.899919 30533 raft_consensus.cc:2781] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:06.900395 30533 raft_consensus.cc:686] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 1 LEADER]: Becoming Leader. State: Replica: 45f6c08513234baeb5ce1582c7b66ed4, State: Running, Role: LEADER
I20260805 22:52:06.900934 30533 consensus_queue.cc:237] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } }
I20260805 22:52:06.906697 29844 catalog_manager.cc:5495] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45f6c08513234baeb5ce1582c7b66ed4 (127.4.28.2). New cstate: current_term: 1 leader_uuid: "45f6c08513234baeb5ce1582c7b66ed4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f6c08513234baeb5ce1582c7b66ed4" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 41333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 } health_report { overall_health: UNKNOWN } } }
W20260805 22:52:06.934152 30496 tablet.cc:2369] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:52:06.939324 30313 tablet.cc:2369] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:52:06.957545 30123 tablet.cc:2369] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:52:06.959852 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:06.954083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.4.28.2:41333)
0805 22:52:06.954292 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:06.954298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:06.954319 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:52:06.954538 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:06.954543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:06.954642 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:06.954789 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:52:06.955210 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.955931 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.957430 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:06.957958 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:06.958092 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:06.958659 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:52:06.959203 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:52:06.959361 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:06.959472 (+ 111us) server_negotiation.cc:299] Negotiation successful
0805 22:52:06.959566 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 22:52:06.969173 30054 raft_consensus.cc:1261] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [term 1 FOLLOWER]: Refusing update from remote peer 45f6c08513234baeb5ce1582c7b66ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:06.969153 30438 raft_consensus.cc:1261] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [term 1 FOLLOWER]: Refusing update from remote peer 45f6c08513234baeb5ce1582c7b66ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:06.970456 30540 consensus_queue.cc:1035] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8973c9f518414c933ab2aba1c289cf" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:06.970979 30533 consensus_queue.cc:1035] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975a7c62511e478cac31ffd4a01a1d20" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:06.984938 30545 mvcc.cc:205] Tried to move back new op lower bound from 7315334459253932032 to 7315334458990968832. Current Snapshot: MvccSnapshot[applied={T|T < 7315334459253932032}]
I20260805 22:52:06.998700 29844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:06.999044 29844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:07.003464 29844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9010c5f31edc45f295ff2a9f89927bcf: Sending DeleteTablet for 3 replicas of tablet 5ae9dbaa53784a58bce67b59cde89046
I20260805 22:52:07.004997 30222 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae9dbaa53784a58bce67b59cde89046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:06 UTC) from {username='slave'} at 127.0.0.1:34256
I20260805 22:52:07.004984 30405 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae9dbaa53784a58bce67b59cde89046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:06 UTC) from {username='slave'} at 127.0.0.1:45716
I20260805 22:52:07.005021 30025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae9dbaa53784a58bce67b59cde89046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:06 UTC) from {username='slave'} at 127.0.0.1:38354
I20260805 22:52:07.008109 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 29930
I20260805 22:52:07.018301 30561 tablet_replica.cc:330] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: stopping tablet replica
I20260805 22:52:07.019443 30562 tablet_replica.cc:330] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: stopping tablet replica
I20260805 22:52:07.019740 30561 raft_consensus.cc:2229] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:07.019927 30562 raft_consensus.cc:2229] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:07.020532 30562 raft_consensus.cc:2258] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:07.020537 30561 raft_consensus.cc:2258] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:07.023221 30562 ts_tablet_manager.cc:1905] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:07.023749 30561 ts_tablet_manager.cc:1905] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:07.026077 30563 tablet_replica.cc:330] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: stopping tablet replica
I20260805 22:52:07.026916 30563 raft_consensus.cc:2229] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:52:07.028066 30563 raft_consensus.cc:2258] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:07.030364 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.009693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.4.28.1:33143)
0805 22:52:07.009897 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:07.009903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:07.013697 (+ 3794us) server_negotiation.cc:407] Connection header received
0805 22:52:07.013938 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:07.013945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:07.014073 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:07.014183 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:07.018701 (+ 4518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:07.019957 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:07.022308 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:07.022981 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:07.023101 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:07.024919 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:07.024932 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:07.024952 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:07.029106 (+ 4154us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:07.029194 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:07.029231 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:07.029396 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:07.029744 (+ 348us) server_negotiation.cc:299] Negotiation successful
0805 22:52:07.029899 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:52:07.030833 30563 ts_tablet_manager.cc:1905] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:07.031575 30562 ts_tablet_manager.cc:1918] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:07.031706 30115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:07.031767 30562 log.cc:1196] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/5ae9dbaa53784a58bce67b59cde89046
I20260805 22:52:07.032371 30562 ts_tablet_manager.cc:1939] T 5ae9dbaa53784a58bce67b59cde89046 P 4f8973c9f518414c933ab2aba1c289cf: Deleting consensus metadata
I20260805 22:52:07.034629 29829 catalog_manager.cc:4841] TS 4f8973c9f518414c933ab2aba1c289cf (127.4.28.3:44655): tablet 5ae9dbaa53784a58bce67b59cde89046 (table test-table [id=2394e6372192487b915cd6c9347b8d3f]) successfully deleted
I20260805 22:52:07.039198 30563 ts_tablet_manager.cc:1918] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:07.039451 30563 log.cc:1196] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/5ae9dbaa53784a58bce67b59cde89046
I20260805 22:52:07.040098 30563 ts_tablet_manager.cc:1939] T 5ae9dbaa53784a58bce67b59cde89046 P 45f6c08513234baeb5ce1582c7b66ed4: Deleting consensus metadata
I20260805 22:52:07.042207 29827 catalog_manager.cc:4841] TS 45f6c08513234baeb5ce1582c7b66ed4 (127.4.28.2:41333): tablet 5ae9dbaa53784a58bce67b59cde89046 (table test-table [id=2394e6372192487b915cd6c9347b8d3f]) successfully deleted
I20260805 22:52:07.141031 30561 ts_tablet_manager.cc:1918] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:07.141331 30561 log.cc:1196] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/5ae9dbaa53784a58bce67b59cde89046
I20260805 22:52:07.142031 30561 ts_tablet_manager.cc:1939] T 5ae9dbaa53784a58bce67b59cde89046 P 975a7c62511e478cac31ffd4a01a1d20: Deleting consensus metadata
I20260805 22:52:07.142009 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 29930
W20260805 22:52:07.175778 29825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.28.1:33143 (error 108)
I20260805 22:52:07.176285 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 30130
I20260805 22:52:07.178113 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.177578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.0.0.1:38374)
0805 22:52:07.177766 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:07.177922 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260805 22:52:07.178391 29825 catalog_manager.cc:4584] TS 975a7c62511e478cac31ffd4a01a1d20 (127.4.28.1:33143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ae9dbaa53784a58bce67b59cde89046: Network error: Client connection negotiation failed: client connection to 127.4.28.1:33143: connect: Connection refused (error 111)
I20260805 22:52:07.185357 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.177236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.4.28.2:41333)
0805 22:52:07.177489 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:07.177496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:07.177540 (+ 44us) server_negotiation.cc:407] Connection header received
0805 22:52:07.177766 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:07.177772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:07.177891 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:07.178013 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:07.178594 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:07.179474 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:07.181473 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:07.182126 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:07.182235 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:07.183499 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:07.183508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:07.183529 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:07.184178 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:07.184237 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:07.184259 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:07.184347 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:07.184540 (+ 193us) server_negotiation.cc:299] Negotiation successful
0805 22:52:07.184669 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260805 22:52:07.186254 30303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:07.232452 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.231807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.0.0.1:38384)
0805 22:52:07.232112 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:07.232236 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260805 22:52:07.233770 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.233291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.0.0.1:38396)
0805 22:52:07.233469 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:07.233600 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260805 22:52:07.294613 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.294001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.0.0.1:38398)
0805 22:52:07.294267 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:07.294402 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260805 22:52:07.295879 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.295451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.0.0.1:38400)
0805 22:52:07.295620 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:07.295713 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260805 22:52:07.305809 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 30130
I20260805 22:52:07.333715 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 30316
I20260805 22:52:07.340859 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.334500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.4.28.3:44655)
0805 22:52:07.334668 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:07.334673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:07.334760 (+ 87us) server_negotiation.cc:407] Connection header received
0805 22:52:07.334917 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:07.334922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:07.335050 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:07.335142 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:07.335598 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:07.336363 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:07.337775 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:07.338218 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:07.338320 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:07.339564 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:07.339575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:07.339592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:07.340211 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:07.340263 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:07.340293 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:07.340385 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:07.340478 (+ 93us) server_negotiation.cc:299] Negotiation successful
0805 22:52:07.340583 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260805 22:52:07.341647 30488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:07.377024 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.376391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.0.0.1:38410)
0805 22:52:07.376690 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:07.376851 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260805 22:52:07.378293 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.377854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.0.0.1:38414)
0805 22:52:07.378070 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:07.378166 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260805 22:52:07.432960 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 30316
I20260805 22:52:07.458388 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 29802
I20260805 22:52:07.459601 29888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:07.575242 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:07.574611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:33143 (local address 127.0.0.1:38430)
0805 22:52:07.574817 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:07.574958 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:33143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260805 22:52:07.575637 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 29802
2026-08-05T22:52:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:07 dist-test-slave-7lbf krb5kdc[30575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:07 dist-test-slave-7lbf krb5kdc[30575](info): set up 2 sockets
Aug 05 22:52:07 dist-test-slave-7lbf krb5kdc[30575](info): commencing operation
krb5kdc: starting...
W20260805 22:52:08.821532 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:09 dist-test-slave-7lbf krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:52:09.172845 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:39613
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:09.536245 30592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:09.536582 30592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:09.536700 30592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:09.544131 30592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:09.544266 30592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:09.544327 30592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:09.544384 30592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:09.544430 30592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:09.554876 30592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:39613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:09.556320 30592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:09.557788 30592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:09.570506 30597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:09.570588 30600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:09.570554 30598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:09.571467 30592 server_base.cc:1034] running on GCE node
I20260805 22:52:09.572314 30592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:09.573923 30592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:09.575143 30592 hybrid_clock.cc:648] HybridClock initialized: now 1785970329575092 us; error 36 us; skew 500 ppm
I20260805 22:52:09.575534 30592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:09 dist-test-slave-7lbf krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970329, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:09.582137 30592 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:52:09.583909 30592 webserver.cc:460] Webserver started at http://127.4.28.62:44949/ using document root <none> and password file <none>
I20260805 22:52:09.584704 30592 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:09.584841 30592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:09.585168 30592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:09.588806 30592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "b0848c1bd5f2445e9b86abe77d880844"
format_stamp: "Formatted at 2026-08-05 22:52:09 on dist-test-slave-7lbf"
server_key: "80e10a5173f074311d455abc32472362"
server_key_iv: "71690f4581191e338f1f5cf6f97cf406"
server_key_version: "encryptionkey@0"
I20260805 22:52:09.589718 30592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "b0848c1bd5f2445e9b86abe77d880844"
format_stamp: "Formatted at 2026-08-05 22:52:09 on dist-test-slave-7lbf"
server_key: "80e10a5173f074311d455abc32472362"
server_key_iv: "71690f4581191e338f1f5cf6f97cf406"
server_key_version: "encryptionkey@0"
I20260805 22:52:09.595687 30592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260805 22:52:09.599871 30610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:09.601413 30592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260805 22:52:09.601645 30592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "b0848c1bd5f2445e9b86abe77d880844"
format_stamp: "Formatted at 2026-08-05 22:52:09 on dist-test-slave-7lbf"
server_key: "80e10a5173f074311d455abc32472362"
server_key_iv: "71690f4581191e338f1f5cf6f97cf406"
server_key_version: "encryptionkey@0"
I20260805 22:52:09.601861 30592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:09.618422 30592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:09.623322 30592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:09.623625 30592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:09.641729 30592 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:39613
I20260805 22:52:09.641753 30681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:39613 every 8 connection(s)
I20260805 22:52:09.643296 30592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:09.647631 30683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:09.647749 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 30592
I20260805 22:52:09.647941 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:09.648373 4208 external_mini_cluster.cc:1428] Setting key aacb207b59da5e1b376f7096186d0948
I20260805 22:52:09.663309 30683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844: Bootstrap starting.
Aug 05 22:52:09 dist-test-slave-7lbf krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:09.667306 30683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:09.668823 30683 log.cc:824] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:09.672616 30683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844: No bootstrap required, opened a new log
I20260805 22:52:09.675292 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:09.650481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.4.28.62:39613)
0805 22:52:09.651476 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:09.651497 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:09.651558 (+ 61us) server_negotiation.cc:407] Connection header received
0805 22:52:09.653367 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:09.653403 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:09.654093 (+ 690us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:09.654639 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:09.656156 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:09.658154 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:09.659748 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:09.660893 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:09.661114 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:09.665904 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:09.665918 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:09.665941 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:09.666017 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:09.670159 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:09.671082 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:09.671093 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:09.671103 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:09.671342 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:09.671740 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:09.671745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:09.671748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:09.672524 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:09.672843 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:09.673349 (+ 506us) server_negotiation.cc:299] Negotiation successful
0805 22:52:09.673854 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":394,"threads_started":1}
I20260805 22:52:09.681945 30683 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0848c1bd5f2445e9b86abe77d880844" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39613 } }
I20260805 22:52:09.682380 30683 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:09.682492 30683 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0848c1bd5f2445e9b86abe77d880844, State: Initialized, Role: FOLLOWER
I20260805 22:52:09.683104 30683 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0848c1bd5f2445e9b86abe77d880844" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39613 } }
I20260805 22:52:09.683346 30683 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:09.683499 30683 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:09.683696 30683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:09.686266 30683 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0848c1bd5f2445e9b86abe77d880844" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39613 } }
I20260805 22:52:09.686791 30683 leader_election.cc:302] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0848c1bd5f2445e9b86abe77d880844; no voters:
I20260805 22:52:09.687413 30683 leader_election.cc:288] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:09.687569 30689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:09.688261 30689 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [term 1 LEADER]: Becoming Leader. State: Replica: b0848c1bd5f2445e9b86abe77d880844, State: Running, Role: LEADER
I20260805 22:52:09.688835 30689 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0848c1bd5f2445e9b86abe77d880844" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39613 } }
I20260805 22:52:09.689675 30683 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:09.692111 30690 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0848c1bd5f2445e9b86abe77d880844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0848c1bd5f2445e9b86abe77d880844" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39613 } } }
I20260805 22:52:09.692085 30695 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0848c1bd5f2445e9b86abe77d880844. Latest consensus state: current_term: 1 leader_uuid: "b0848c1bd5f2445e9b86abe77d880844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0848c1bd5f2445e9b86abe77d880844" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39613 } } }
I20260805 22:52:09.692565 30690 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:09.692574 30695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:09.693887 30700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:09.699381 30700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:09.709198 30700 catalog_manager.cc:1347] Generated new cluster ID: ca9830da5afe488a981a2c0e547bbfc2
I20260805 22:52:09.709337 30700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:09.737984 30700 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:09.740130 30700 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:09.761276 30700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844: Generated new TSK 0
I20260805 22:52:09.762535 30700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:52:09.848634 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39613
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:10.221108 30722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:10.221468 30722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:10.221629 30722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:10.229357 30722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:10.229492 30722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:10.229664 30722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:52:10.240137 30722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:10.241642 30722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:10.243175 30722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:10.257342 30728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:10.257277 30731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:10.257370 30727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:10.258198 30722 server_base.cc:1034] running on GCE node
I20260805 22:52:10.258844 30722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:10.260012 30722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:10.261233 30722 hybrid_clock.cc:648] HybridClock initialized: now 1785970330261176 us; error 50 us; skew 500 ppm
I20260805 22:52:10.261643 30722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:10 dist-test-slave-7lbf krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970330, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:10.265239 30722 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:52:10.267035 30722 webserver.cc:460] Webserver started at http://127.4.28.1:32807/ using document root <none> and password file <none>
I20260805 22:52:10.267851 30722 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:10.267987 30722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:10.268328 30722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:10.271617 30722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "11f4e61f43764087a0667c70fa7938bf"
format_stamp: "Formatted at 2026-08-05 22:52:10 on dist-test-slave-7lbf"
server_key: "7535bec998d7fbc5e73ad40e8ed2f349"
server_key_iv: "bb434a7292ae3f2698a6e0dd8021ce73"
server_key_version: "encryptionkey@0"
I20260805 22:52:10.272495 30722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "11f4e61f43764087a0667c70fa7938bf"
format_stamp: "Formatted at 2026-08-05 22:52:10 on dist-test-slave-7lbf"
server_key: "7535bec998d7fbc5e73ad40e8ed2f349"
server_key_iv: "bb434a7292ae3f2698a6e0dd8021ce73"
server_key_version: "encryptionkey@0"
I20260805 22:52:10.278301 30722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260805 22:52:10.282369 30739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:10.283882 30722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:52:10.284083 30722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "11f4e61f43764087a0667c70fa7938bf"
format_stamp: "Formatted at 2026-08-05 22:52:10 on dist-test-slave-7lbf"
server_key: "7535bec998d7fbc5e73ad40e8ed2f349"
server_key_iv: "bb434a7292ae3f2698a6e0dd8021ce73"
server_key_version: "encryptionkey@0"
I20260805 22:52:10.284300 30722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:10.302906 30722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:10.307698 30722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:10.308010 30722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:10.309122 30722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:10.310693 30722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:10.310782 30722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:10.310925 30722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:10.310997 30722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:10.344045 30722 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:42571
I20260805 22:52:10.344111 30887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:42571 every 8 connection(s)
I20260805 22:52:10.345468 30722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:52:10.355859 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 30722
I20260805 22:52:10.356038 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:52:10.356484 4208 external_mini_cluster.cc:1428] Setting key 5f1f94e3b2fdd1efcd10fe24a4f8d963
Aug 05 22:52:10 dist-test-slave-7lbf krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970330, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:10.368491 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:10.348359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:44493 (local address 127.4.28.62:39613)
0805 22:52:10.348865 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:10.348872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:10.349945 (+ 1073us) server_negotiation.cc:407] Connection header received
0805 22:52:10.351952 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:10.351958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:10.352053 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:10.352161 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:10.353896 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:10.354719 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:10.356561 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:10.357020 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:10.357168 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:10.361635 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:10.361642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:10.361647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:10.361695 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:10.364818 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:10.365800 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:10.365806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:10.365809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:10.365987 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:10.366527 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:10.366532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:10.366535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:10.366895 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:10.367044 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:10.367849 (+ 805us) server_negotiation.cc:299] Negotiation successful
0805 22:52:10.368120 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":227,"threads_started":1}
I20260805 22:52:10.369341 30894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:10.348868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:39613 (local address 127.4.28.1:44493)
0805 22:52:10.349771 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:10.349816 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:10.351631 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:10.352396 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:10.352405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:10.352983 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:10.353692 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:10.353716 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:10.354870 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:10.354874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:10.356376 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:10.356392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:10.357308 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:10.357315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:10.357688 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:10.358417 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:52:10.358449 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:10.361284 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:10.365025 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:10.365036 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:10.365053 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:10.365646 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:10.366131 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:10.366137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:10.366141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:10.366397 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:10.367029 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:10.367036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:10.367419 (+ 383us) client_negotiation.cc:769] Sending connection context
0805 22:52:10.367684 (+ 265us) client_negotiation.cc:240] Negotiation successful
0805 22:52:10.368149 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":353,"threads_started":1}
I20260805 22:52:10.371114 30889 heartbeater.cc:347] Connected to a master server at 127.4.28.62:39613
I20260805 22:52:10.371529 30889 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:10.372524 30889 heartbeater.cc:511] Master 127.4.28.62:39613 requested a full tablet report, sending...
I20260805 22:52:10.375156 30635 ts_manager.cc:194] Registered new tserver with Master: 11f4e61f43764087a0667c70fa7938bf (127.4.28.1:42571)
I20260805 22:52:10.377096 30635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:44493
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:52:10.435279 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39613
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:10.798653 30898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:10.798969 30898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:10.799074 30898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:10.806564 30898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:10.806692 30898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:10.806808 30898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:52:10.817061 30898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:10.818531 30898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:10.819906 30898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:10.833707 30908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:10.833791 30909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:10.833746 30912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:10.834609 30898 server_base.cc:1034] running on GCE node
I20260805 22:52:10.835228 30898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:10.836221 30898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:10.837474 30898 hybrid_clock.cc:648] HybridClock initialized: now 1785970330837415 us; error 42 us; skew 500 ppm
I20260805 22:52:10.837852 30898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:10 dist-test-slave-7lbf krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970330, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:10.841126 30898 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:52:10.842839 30898 webserver.cc:460] Webserver started at http://127.4.28.2:46419/ using document root <none> and password file <none>
I20260805 22:52:10.843571 30898 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:10.843679 30898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:10.843981 30898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:10.847332 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "09f69cd40ce646078abe383d2689b165"
format_stamp: "Formatted at 2026-08-05 22:52:10 on dist-test-slave-7lbf"
server_key: "09e913761c0a7ea7e3746eed26b9cc2c"
server_key_iv: "0caadca92592097bec1aad0ab21a84e6"
server_key_version: "encryptionkey@0"
I20260805 22:52:10.848181 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "09f69cd40ce646078abe383d2689b165"
format_stamp: "Formatted at 2026-08-05 22:52:10 on dist-test-slave-7lbf"
server_key: "09e913761c0a7ea7e3746eed26b9cc2c"
server_key_iv: "0caadca92592097bec1aad0ab21a84e6"
server_key_version: "encryptionkey@0"
I20260805 22:52:10.854293 30898 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260805 22:52:10.858269 30921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:10.859838 30898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:52:10.860026 30898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "09f69cd40ce646078abe383d2689b165"
format_stamp: "Formatted at 2026-08-05 22:52:10 on dist-test-slave-7lbf"
server_key: "09e913761c0a7ea7e3746eed26b9cc2c"
server_key_iv: "0caadca92592097bec1aad0ab21a84e6"
server_key_version: "encryptionkey@0"
I20260805 22:52:10.860234 30898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:10.879863 30898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:10.885401 30898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:10.885780 30898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:10.886938 30898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:10.888685 30898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:10.888778 30898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:10.888878 30898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:10.888922 30898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:10.922845 30898 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:34613
I20260805 22:52:10.922960 31076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:34613 every 8 connection(s)
I20260805 22:52:10.924314 30898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:52:10.931571 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 30898
I20260805 22:52:10.931730 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:52:10.932229 4208 external_mini_cluster.cc:1428] Setting key 23c3395c3620548dc95e44c70c93e606
Aug 05 22:52:10 dist-test-slave-7lbf krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970330, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:10.946569 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:10.927346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:47343 (local address 127.4.28.62:39613)
0805 22:52:10.927840 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:10.927845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:10.928860 (+ 1015us) server_negotiation.cc:407] Connection header received
0805 22:52:10.930923 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:10.930929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:10.931021 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:10.931154 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:10.932950 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:10.933842 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:10.935665 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:10.936136 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:10.936258 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:10.940234 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:10.940241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:10.940244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:10.940286 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:10.943081 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:10.943938 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:10.943943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:10.943946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:10.944092 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:10.944568 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:10.944572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:10.944574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:10.944901 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:10.945058 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:10.945908 (+ 850us) server_negotiation.cc:299] Negotiation successful
0805 22:52:10.946167 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260805 22:52:10.947705 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:10.927842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:39613 (local address 127.4.28.2:47343)
0805 22:52:10.928657 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:10.928704 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:10.930582 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:10.931427 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:10.931437 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:10.932072 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:10.932742 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:10.932763 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:10.934033 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:10.934037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:10.935491 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:10.935508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:10.936421 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:10.936430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:10.936660 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:10.937336 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:52:10.937363 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:10.939929 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:10.943269 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:10.943280 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:10.943297 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:10.943793 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:10.944216 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:10.944221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:10.944224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:10.944447 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:10.945069 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:10.945078 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:10.945475 (+ 397us) client_negotiation.cc:769] Sending connection context
0805 22:52:10.945815 (+ 340us) client_negotiation.cc:240] Negotiation successful
0805 22:52:10.946446 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":312,"threads_started":1}
I20260805 22:52:10.949822 31077 heartbeater.cc:347] Connected to a master server at 127.4.28.62:39613
I20260805 22:52:10.950171 31077 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:10.950976 31077 heartbeater.cc:511] Master 127.4.28.62:39613 requested a full tablet report, sending...
I20260805 22:52:10.952740 30635 ts_manager.cc:194] Registered new tserver with Master: 09f69cd40ce646078abe383d2689b165 (127.4.28.2:34613)
I20260805 22:52:10.953835 30635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:47343
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:52:11.009881 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39613
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:11.377054 31086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:11.377388 31086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:11.377490 31086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260805 22:52:11.380803 30889 heartbeater.cc:503] Master 127.4.28.62:39613 was elected leader, sending a full tablet report...
W20260805 22:52:11.385349 31086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:11.385540 31086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:11.385712 31086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:52:11.396265 31086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:11.397714 31086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:11.399154 31086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:11.413697 31092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:11.413738 31095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:11.413661 31093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:11.414739 31086 server_base.cc:1034] running on GCE node
I20260805 22:52:11.415514 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:11.416662 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:11.417922 31086 hybrid_clock.cc:648] HybridClock initialized: now 1785970331417864 us; error 54 us; skew 500 ppm
I20260805 22:52:11.418298 31086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:11 dist-test-slave-7lbf krb5kdc[30575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:11.421934 31086 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:52:11.423597 31086 webserver.cc:460] Webserver started at http://127.4.28.3:38609/ using document root <none> and password file <none>
I20260805 22:52:11.424350 31086 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:11.424480 31086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:11.424805 31086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:11.428155 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "7fc0bb635ba24f24afe9c8b40c72a86e"
format_stamp: "Formatted at 2026-08-05 22:52:11 on dist-test-slave-7lbf"
server_key: "06d8057156897b760cc8b5f06d5e0ed1"
server_key_iv: "492d260e4e5c0c6128b7e737e3d29a6f"
server_key_version: "encryptionkey@0"
I20260805 22:52:11.429039 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "7fc0bb635ba24f24afe9c8b40c72a86e"
format_stamp: "Formatted at 2026-08-05 22:52:11 on dist-test-slave-7lbf"
server_key: "06d8057156897b760cc8b5f06d5e0ed1"
server_key_iv: "492d260e4e5c0c6128b7e737e3d29a6f"
server_key_version: "encryptionkey@0"
I20260805 22:52:11.435016 31086 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260805 22:52:11.439164 31104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:11.440716 31086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260805 22:52:11.440969 31086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "7fc0bb635ba24f24afe9c8b40c72a86e"
format_stamp: "Formatted at 2026-08-05 22:52:11 on dist-test-slave-7lbf"
server_key: "06d8057156897b760cc8b5f06d5e0ed1"
server_key_iv: "492d260e4e5c0c6128b7e737e3d29a6f"
server_key_version: "encryptionkey@0"
I20260805 22:52:11.441201 31086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:11.461205 31086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:11.466131 31086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:11.466434 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:11.467581 31086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:11.469151 31086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:11.469237 31086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:11.469368 31086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:11.469424 31086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:11.501621 31086 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:42555
I20260805 22:52:11.501694 31251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:42555 every 8 connection(s)
I20260805 22:52:11.503013 31086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:52:11.508327 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 31086
I20260805 22:52:11.508499 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:52:11.508952 4208 external_mini_cluster.cc:1428] Setting key 2cf22f5b7ca3515c26e29fda477424fb
Aug 05 22:52:11 dist-test-slave-7lbf krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:11.524288 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.505819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:41121 (local address 127.4.28.62:39613)
0805 22:52:11.506295 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:11.506301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:11.507469 (+ 1168us) server_negotiation.cc:407] Connection header received
0805 22:52:11.509487 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:11.509493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:11.509624 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:11.509726 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:11.511402 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.512223 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.513891 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.514302 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.514421 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.518194 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:11.518200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:11.518204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:11.518251 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:11.520860 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:11.521716 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:11.521727 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:11.521730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:11.521876 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:11.522352 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:11.522356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:11.522358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:11.522675 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:11.522821 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:11.523640 (+ 819us) server_negotiation.cc:299] Negotiation successful
0805 22:52:11.523898 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260805 22:52:11.525298 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.506491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:39613 (local address 127.4.28.3:41121)
0805 22:52:11.507290 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:11.507332 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:11.509175 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:11.510024 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:11.510034 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:11.510607 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:11.511221 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.511241 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.512386 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.512390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.513734 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.513749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.514557 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.514566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.514799 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.515486 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:52:11.515520 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:11.517918 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:11.521042 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:11.521052 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:11.521068 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:11.521575 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:11.521997 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:11.522003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:11.522007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:11.522225 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:11.522816 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:11.522826 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:11.523221 (+ 395us) client_negotiation.cc:769] Sending connection context
0805 22:52:11.523488 (+ 267us) client_negotiation.cc:240] Negotiation successful
0805 22:52:11.524113 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":304,"threads_started":1}
I20260805 22:52:11.526803 31252 heartbeater.cc:347] Connected to a master server at 127.4.28.62:39613
I20260805 22:52:11.527143 31252 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:11.528002 31252 heartbeater.cc:511] Master 127.4.28.62:39613 requested a full tablet report, sending...
I20260805 22:52:11.529479 30635 ts_manager.cc:194] Registered new tserver with Master: 7fc0bb635ba24f24afe9c8b40c72a86e (127.4.28.3:42555)
I20260805 22:52:11.530383 30635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:41121
I20260805 22:52:11.537101 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:52:11.555558 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.541666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.4.28.62:39613)
0805 22:52:11.541841 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:11.541847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:11.542300 (+ 453us) server_negotiation.cc:407] Connection header received
0805 22:52:11.542484 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:11.542488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:11.542576 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:11.542659 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:11.544157 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.544898 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.546342 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.546802 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.546904 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.549718 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:11.549723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:11.549727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:11.549767 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:11.552436 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:11.553446 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:11.553451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:11.553453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:11.553656 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:11.554116 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:11.554120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:11.554122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:11.554483 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:11.554711 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:11.555148 (+ 437us) server_negotiation.cc:299] Negotiation successful
0805 22:52:11.555312 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 22:52:11.560222 30635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:52:11.568017 30635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:52:11.609092 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.581820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42571 (local address 127.0.0.1:44258)
0805 22:52:11.582960 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:11.583014 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:11.583234 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:11.599739 (+ 16505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:11.599749 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:11.599792 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:11.600364 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.600383 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.603091 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.603099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.605869 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.605888 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.607955 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.607969 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.608226 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.608289 (+ 63us) client_negotiation.cc:769] Sending connection context
0805 22:52:11.608418 (+ 129us) client_negotiation.cc:240] Negotiation successful
0805 22:52:11.608653 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":864,"threads_started":1}
I20260805 22:52:11.609966 31272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.582186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.4.28.1:42571)
0805 22:52:11.598764 (+ 16578us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:11.598776 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:11.598823 (+ 47us) server_negotiation.cc:407] Connection header received
0805 22:52:11.598968 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:11.598979 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:11.599334 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:11.600695 (+ 1361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:11.601011 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.602855 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.606096 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.607459 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.607611 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.607820 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:11.609404 (+ 1584us) server_negotiation.cc:299] Negotiation successful
0805 22:52:11.609642 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16363,"thread_start_us":16178,"threads_started":1}
I20260805 22:52:11.613631 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.582179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:42555 (local address 127.0.0.1:60734)
0805 22:52:11.604344 (+ 22165us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:11.604367 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:11.604530 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:11.606236 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:11.606241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:11.606267 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:11.606667 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.606683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.608811 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.608816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.610853 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.610867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.612744 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.612751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.612929 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.612967 (+ 38us) client_negotiation.cc:769] Sending connection context
0805 22:52:11.613077 (+ 110us) client_negotiation.cc:240] Negotiation successful
0805 22:52:11.613224 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22012,"thread_start_us":23136,"threads_started":1}
I20260805 22:52:11.613703 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.582413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.4.28.3:42555)
0805 22:52:11.605254 (+ 22841us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:11.605265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:11.605306 (+ 41us) server_negotiation.cc:407] Connection header received
0805 22:52:11.605420 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:11.605429 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:11.605814 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:11.606066 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:11.606985 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.608648 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.611050 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.612522 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.612663 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.612909 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:11.613135 (+ 226us) server_negotiation.cc:299] Negotiation successful
0805 22:52:11.613306 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22624,"thread_start_us":22405,"threads_started":1}
I20260805 22:52:11.614804 31276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.584343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:34613 (local address 127.0.0.1:55370)
0805 22:52:11.605784 (+ 21441us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:11.605804 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:11.605954 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:11.606594 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:11.606602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:11.606632 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:11.608153 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.608167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.610300 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.610305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.612023 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.612038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.614045 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.614056 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.614227 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.614260 (+ 33us) client_negotiation.cc:769] Sending connection context
0805 22:52:11.614345 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 22:52:11.614481 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21307,"thread_start_us":19663,"threads_started":1}
I20260805 22:52:11.614635 30802 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa0a441ab0149dcb90d6702c2ec524d (DEFAULT_TABLE table=test-table [id=659f1166c7014400af546a0e31c63f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:11.616284 30802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa0a441ab0149dcb90d6702c2ec524d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:11.617869 31166 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa0a441ab0149dcb90d6702c2ec524d (DEFAULT_TABLE table=test-table [id=659f1166c7014400af546a0e31c63f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:11.619076 31166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa0a441ab0149dcb90d6702c2ec524d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:11.615092 31284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.584785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.4.28.2:34613)
0805 22:52:11.603545 (+ 18760us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:11.603555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:11.605946 (+ 2391us) server_negotiation.cc:407] Connection header received
0805 22:52:11.606120 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:11.606128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:11.606365 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:11.606587 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:11.608383 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.610108 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.612197 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.613545 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.613680 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.613880 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:11.614434 (+ 554us) server_negotiation.cc:299] Negotiation successful
0805 22:52:11.614615 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18531,"thread_start_us":18773,"threads_started":1}
I20260805 22:52:11.628557 31285 tablet_bootstrap.cc:492] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: Bootstrap starting.
I20260805 22:52:11.629436 30989 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa0a441ab0149dcb90d6702c2ec524d (DEFAULT_TABLE table=test-table [id=659f1166c7014400af546a0e31c63f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:11.631183 30989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa0a441ab0149dcb90d6702c2ec524d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:11.632277 31287 tablet_bootstrap.cc:492] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: Bootstrap starting.
I20260805 22:52:11.632449 31285 tablet_bootstrap.cc:654] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:11.633916 31285 log.cc:824] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:11.636085 31287 tablet_bootstrap.cc:654] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:11.637457 31287 log.cc:824] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:11.638649 31285 tablet_bootstrap.cc:492] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: No bootstrap required, opened a new log
I20260805 22:52:11.639006 31285 ts_tablet_manager.cc:1397] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260805 22:52:11.641284 31287 tablet_bootstrap.cc:492] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: No bootstrap required, opened a new log
I20260805 22:52:11.641748 31287 ts_tablet_manager.cc:1397] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260805 22:52:11.650364 31285 raft_consensus.cc:348] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } }
I20260805 22:52:11.652343 31285 raft_consensus.cc:374] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:11.652568 31285 raft_consensus.cc:729] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f4e61f43764087a0667c70fa7938bf, State: Initialized, Role: FOLLOWER
I20260805 22:52:11.652858 31287 raft_consensus.cc:348] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } }
I20260805 22:52:11.653455 31287 raft_consensus.cc:374] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:11.653661 31287 raft_consensus.cc:729] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc0bb635ba24f24afe9c8b40c72a86e, State: Initialized, Role: FOLLOWER
I20260805 22:52:11.653456 31285 consensus_queue.cc:260] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } }
I20260805 22:52:11.654608 31287 consensus_queue.cc:260] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } }
I20260805 22:52:11.654918 31291 tablet_bootstrap.cc:492] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: Bootstrap starting.
I20260805 22:52:11.658777 31285 ts_tablet_manager.cc:1428] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: Time spent starting tablet: real 0.020s user 0.015s sys 0.000s
I20260805 22:52:11.659675 31291 tablet_bootstrap.cc:654] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:11.660567 31252 heartbeater.cc:503] Master 127.4.28.62:39613 was elected leader, sending a full tablet report...
I20260805 22:52:11.661314 31291 log.cc:824] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:11.661432 31287 ts_tablet_manager.cc:1428] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: Time spent starting tablet: real 0.019s user 0.016s sys 0.000s
I20260805 22:52:11.666728 31291 tablet_bootstrap.cc:492] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: No bootstrap required, opened a new log
I20260805 22:52:11.667166 31291 ts_tablet_manager.cc:1397] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260805 22:52:11.675099 31291 raft_consensus.cc:348] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } }
I20260805 22:52:11.675542 31291 raft_consensus.cc:374] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:11.675652 31291 raft_consensus.cc:729] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f69cd40ce646078abe383d2689b165, State: Initialized, Role: FOLLOWER
I20260805 22:52:11.676350 31291 consensus_queue.cc:260] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } }
I20260805 22:52:11.677707 31077 heartbeater.cc:503] Master 127.4.28.62:39613 was elected leader, sending a full tablet report...
I20260805 22:52:11.678205 31291 ts_tablet_manager.cc:1428] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260805 22:52:11.754905 31253 tablet.cc:2369] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:52:11.801008 31293 raft_consensus.cc:482] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:52:11.801378 31293 raft_consensus.cc:504] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } }
I20260805 22:52:11.803246 31293 leader_election.cc:288] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11f4e61f43764087a0667c70fa7938bf (127.4.28.1:42571), 09f69cd40ce646078abe383d2689b165 (127.4.28.2:34613)
I20260805 22:52:11.809290 31272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.803679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:54767 (local address 127.4.28.1:42571)
0805 22:52:11.803902 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:11.803908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:11.804033 (+ 125us) server_negotiation.cc:407] Connection header received
0805 22:52:11.804236 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:11.804240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:11.804305 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:11.804380 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:11.804833 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.805633 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.807409 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.808384 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.808513 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.808619 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:11.808901 (+ 282us) server_negotiation.cc:299] Negotiation successful
0805 22:52:11.809044 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 22:52:11.809321 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.803513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42571 (local address 127.4.28.3:54767)
0805 22:52:11.803902 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:11.803922 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:11.804084 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:11.804447 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:11.804452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:11.804496 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:11.804681 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.804690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.805789 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.805793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.807228 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.807243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.808554 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.808560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.808733 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.808760 (+ 27us) client_negotiation.cc:769] Sending connection context
0805 22:52:11.808852 (+ 92us) client_negotiation.cc:240] Negotiation successful
0805 22:52:11.808997 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":19}
I20260805 22:52:11.810088 31284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.803764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:55919 (local address 127.4.28.2:34613)
0805 22:52:11.803995 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:11.804002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:11.804713 (+ 711us) server_negotiation.cc:407] Connection header received
0805 22:52:11.804886 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:11.804890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:11.804951 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:11.805030 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:11.805656 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.806473 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.808280 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.809074 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.809200 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.809299 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:11.809706 (+ 407us) server_negotiation.cc:299] Negotiation successful
0805 22:52:11.809854 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260805 22:52:11.810097 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.803696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:34613 (local address 127.4.28.3:55919)
0805 22:52:11.804578 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:11.804594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:11.804744 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:11.805063 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:11.805067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:11.805092 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:11.805462 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.805474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.806627 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.806633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.808096 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:11.808110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.809312 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.809320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:11.809507 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.809566 (+ 59us) client_negotiation.cc:769] Sending connection context
0805 22:52:11.809655 (+ 89us) client_negotiation.cc:240] Negotiation successful
0805 22:52:11.809791 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":4,"thread_start_us":493,"threads_started":1}
I20260805 22:52:11.810626 30832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa0a441ab0149dcb90d6702c2ec524d" candidate_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4e61f43764087a0667c70fa7938bf" is_pre_election: true
I20260805 22:52:11.811164 30832 raft_consensus.cc:2454] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fc0bb635ba24f24afe9c8b40c72a86e in term 0.
I20260805 22:52:11.811098 31013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa0a441ab0149dcb90d6702c2ec524d" candidate_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f69cd40ce646078abe383d2689b165" is_pre_election: true
I20260805 22:52:11.811548 31013 raft_consensus.cc:2454] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fc0bb635ba24f24afe9c8b40c72a86e in term 0.
I20260805 22:52:11.811874 31111 leader_election.cc:302] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f4e61f43764087a0667c70fa7938bf, 7fc0bb635ba24f24afe9c8b40c72a86e; no voters:
I20260805 22:52:11.812317 31293 raft_consensus.cc:2781] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:52:11.812446 31293 raft_consensus.cc:482] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:52:11.812552 31293 raft_consensus.cc:3037] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:11.815059 31293 raft_consensus.cc:504] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } }
I20260805 22:52:11.815758 31293 leader_election.cc:288] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [CANDIDATE]: Term 1 election: Requested vote from peers 11f4e61f43764087a0667c70fa7938bf (127.4.28.1:42571), 09f69cd40ce646078abe383d2689b165 (127.4.28.2:34613)
I20260805 22:52:11.816459 30832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa0a441ab0149dcb90d6702c2ec524d" candidate_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f4e61f43764087a0667c70fa7938bf"
I20260805 22:52:11.816458 31013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa0a441ab0149dcb90d6702c2ec524d" candidate_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f69cd40ce646078abe383d2689b165"
I20260805 22:52:11.816674 30832 raft_consensus.cc:3037] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:11.816700 31013 raft_consensus.cc:3037] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:11.819101 30832 raft_consensus.cc:2454] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fc0bb635ba24f24afe9c8b40c72a86e in term 1.
I20260805 22:52:11.819168 31013 raft_consensus.cc:2454] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fc0bb635ba24f24afe9c8b40c72a86e in term 1.
I20260805 22:52:11.819653 31111 leader_election.cc:302] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f4e61f43764087a0667c70fa7938bf, 7fc0bb635ba24f24afe9c8b40c72a86e; no voters:
I20260805 22:52:11.819990 31293 raft_consensus.cc:2781] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:11.820484 31293 raft_consensus.cc:686] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 1 LEADER]: Becoming Leader. State: Replica: 7fc0bb635ba24f24afe9c8b40c72a86e, State: Running, Role: LEADER
I20260805 22:52:11.821019 31293 consensus_queue.cc:237] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } }
I20260805 22:52:11.826270 30635 catalog_manager.cc:5495] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fc0bb635ba24f24afe9c8b40c72a86e (127.4.28.3). New cstate: current_term: 1 leader_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fc0bb635ba24f24afe9c8b40c72a86e" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 42555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 } health_report { overall_health: UNKNOWN } } }
W20260805 22:52:11.848304 30891 tablet.cc:2369] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:52:11.896862 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.891315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.4.28.3:42555)
0805 22:52:11.891547 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:11.891552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:11.891571 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:52:11.891752 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:11.891756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:11.891846 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:11.891986 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:52:11.892429 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.893179 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.894631 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.895034 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.895131 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.895676 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:52:11.896210 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:52:11.896346 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:11.896527 (+ 181us) server_negotiation.cc:299] Negotiation successful
0805 22:52:11.896577 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:52:11.905727 31013 raft_consensus.cc:1261] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [term 1 FOLLOWER]: Refusing update from remote peer 7fc0bb635ba24f24afe9c8b40c72a86e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:11.905727 30832 raft_consensus.cc:1261] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [term 1 FOLLOWER]: Refusing update from remote peer 7fc0bb635ba24f24afe9c8b40c72a86e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:11.906773 31293 consensus_queue.cc:1035] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f4e61f43764087a0667c70fa7938bf" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:11.907168 31298 consensus_queue.cc:1035] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f69cd40ce646078abe383d2689b165" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 34613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:11.924903 31303 mvcc.cc:205] Tried to move back new op lower bound from 7315334479475052544 to 7315334479143288832. Current Snapshot: MvccSnapshot[applied={T|T < 7315334479475052544}]
I20260805 22:52:11.936566 30631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:11.936940 30631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:11.942014 30631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0848c1bd5f2445e9b86abe77d880844: Sending DeleteTablet for 3 replicas of tablet 4aa0a441ab0149dcb90d6702c2ec524d
I20260805 22:52:11.943593 31166 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa0a441ab0149dcb90d6702c2ec524d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:11 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:60734
I20260805 22:52:11.943670 30802 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa0a441ab0149dcb90d6702c2ec524d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:11 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:44258
I20260805 22:52:11.944994 30989 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa0a441ab0149dcb90d6702c2ec524d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:11 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:55370
I20260805 22:52:11.947011 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 30722
I20260805 22:52:11.954147 31317 tablet_replica.cc:330] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: stopping tablet replica
I20260805 22:52:11.955543 31317 raft_consensus.cc:2229] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:11.956619 31316 tablet_replica.cc:330] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: stopping tablet replica
I20260805 22:52:11.957463 31317 raft_consensus.cc:2258] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:11.958124 31316 raft_consensus.cc:2229] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:52:11.959335 31316 raft_consensus.cc:2258] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:11.960011 31318 tablet_replica.cc:330] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: stopping tablet replica
I20260805 22:52:11.960507 31318 raft_consensus.cc:2229] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:11.960939 31318 raft_consensus.cc:2258] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:11.963449 31317 ts_tablet_manager.cc:1905] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:11.969467 31316 ts_tablet_manager.cc:1905] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:11.971591 31317 ts_tablet_manager.cc:1918] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:11.971940 31317 log.cc:1196] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/4aa0a441ab0149dcb90d6702c2ec524d
I20260805 22:52:11.972426 31318 ts_tablet_manager.cc:1905] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:11.972723 31317 ts_tablet_manager.cc:1939] T 4aa0a441ab0149dcb90d6702c2ec524d P 11f4e61f43764087a0667c70fa7938bf: Deleting consensus metadata
I20260805 22:52:11.974895 30615 catalog_manager.cc:4841] TS 11f4e61f43764087a0667c70fa7938bf (127.4.28.1:42571): tablet 4aa0a441ab0149dcb90d6702c2ec524d (table test-table [id=659f1166c7014400af546a0e31c63f68]) successfully deleted
Aug 05 22:52:11 dist-test-slave-7lbf krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:52:11.980571 31316 ts_tablet_manager.cc:1918] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:11.980855 31316 log.cc:1196] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/4aa0a441ab0149dcb90d6702c2ec524d
I20260805 22:52:11.981536 31316 ts_tablet_manager.cc:1939] T 4aa0a441ab0149dcb90d6702c2ec524d P 7fc0bb635ba24f24afe9c8b40c72a86e: Deleting consensus metadata
I20260805 22:52:11.983821 30612 catalog_manager.cc:4841] TS 7fc0bb635ba24f24afe9c8b40c72a86e (127.4.28.3:42555): tablet 4aa0a441ab0149dcb90d6702c2ec524d (table test-table [id=659f1166c7014400af546a0e31c63f68]) successfully deleted
I20260805 22:52:11.985281 31318 ts_tablet_manager.cc:1918] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:11.985502 31318 log.cc:1196] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/4aa0a441ab0149dcb90d6702c2ec524d
I20260805 22:52:11.986042 31318 ts_tablet_manager.cc:1939] T 4aa0a441ab0149dcb90d6702c2ec524d P 09f69cd40ce646078abe383d2689b165: Deleting consensus metadata
I20260805 22:52:11.987687 31272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:11.948535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.4.28.1:42571)
0805 22:52:11.948750 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:11.948758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:11.967640 (+ 18882us) server_negotiation.cc:407] Connection header received
0805 22:52:11.968161 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:11.968168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:11.968261 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:11.968357 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:11.970385 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.971499 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.973504 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:11.974210 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:11.974315 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:11.979905 (+ 5590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:11.979929 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:11.979949 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:11.980023 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:11.983929 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:11.984960 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:11.984972 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:11.984976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:11.985170 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:11.985766 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:11.985773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:11.985777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:11.986313 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:11.986653 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:11.987030 (+ 377us) server_negotiation.cc:299] Negotiation successful
0805 22:52:11.987290 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:52:11.988144 30611 catalog_manager.cc:4841] TS 09f69cd40ce646078abe383d2689b165 (127.4.28.2:34613): tablet 4aa0a441ab0149dcb90d6702c2ec524d (table test-table [id=659f1166c7014400af546a0e31c63f68]) successfully deleted
I20260805 22:52:11.988629 30882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:12.086308 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 30722
I20260805 22:52:12.115091 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 30898
Aug 05 22:52:12 dist-test-slave-7lbf krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:52:12.134162 31284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:12.116061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.4.28.2:34613)
0805 22:52:12.116303 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:12.116310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:12.116337 (+ 27us) server_negotiation.cc:407] Connection header received
0805 22:52:12.116411 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:12.116417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:12.116505 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:12.116624 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:12.118203 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:12.119442 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:12.120985 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:12.121810 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:12.121957 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:12.126787 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:12.126812 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:12.126835 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:12.126891 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:12.130767 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:12.131734 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:12.131745 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:12.131748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:12.131920 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:12.132311 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:12.132316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:12.132319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:12.132845 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:12.133171 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:12.133540 (+ 369us) server_negotiation.cc:299] Negotiation successful
0805 22:52:12.133785 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260805 22:52:12.135033 31068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:12.226987 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 30898
I20260805 22:52:12.251585 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 31086
Aug 05 22:52:12 dist-test-slave-7lbf krb5kdc[30575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:52:12.269066 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:12.252456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.4.28.3:42555)
0805 22:52:12.252687 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:12.252694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:12.252787 (+ 93us) server_negotiation.cc:407] Connection header received
0805 22:52:12.252991 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:12.252998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:12.253090 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:12.253183 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:12.254799 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:12.255698 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:12.257123 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:12.257672 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:12.257803 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:12.262471 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:12.262486 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:12.262509 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:12.262559 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:12.265908 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:12.266772 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:12.266780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:12.266782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:12.266965 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:12.267362 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:12.267366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:12.267368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:12.267775 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:12.268122 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:12.268427 (+ 305us) server_negotiation.cc:299] Negotiation successful
0805 22:52:12.268674 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260805 22:52:12.270044 31244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:12.397677 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 31086
I20260805 22:52:12.425262 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 30592
I20260805 22:52:12.426697 30675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:12.544071 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 30592
2026-08-05T22:52:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4973 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-05T22:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:12Z Disabled control of system clock
I20260805 22:52:12.651821 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44007
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:44007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 22:52:13.028023 31329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:13.028360 31329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:13.028452 31329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:13.036159 31329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:13.036302 31329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:13.036342 31329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:13.036393 31329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:13.036439 31329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:13.046275 31329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:44007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:13.048625 31329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:13.050879 31329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:13.063879 31334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:13.063827 31336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:13.063891 31341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:13.065002 31329 server_base.cc:1034] running on GCE node
I20260805 22:52:13.066054 31329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:13.067756 31329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:13.069024 31329 hybrid_clock.cc:648] HybridClock initialized: now 1785970333068949 us; error 69 us; skew 500 ppm
I20260805 22:52:13.069435 31329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:13.072124 31329 webserver.cc:460] Webserver started at http://127.4.28.62:45499/ using document root <none> and password file <none>
I20260805 22:52:13.072928 31329 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:13.073051 31329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:13.073400 31329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:13.076653 31329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "638fd766a87a4c7a837cded76c9f23c9"
format_stamp: "Formatted at 2026-08-05 22:52:13 on dist-test-slave-7lbf"
server_key: "0b4cb74614f2b6b54bcce02a9110cc19"
server_key_iv: "4d92180fbdcee35fdd369fa66d133381"
server_key_version: "encryptionkey@0"
I20260805 22:52:13.077596 31329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "638fd766a87a4c7a837cded76c9f23c9"
format_stamp: "Formatted at 2026-08-05 22:52:13 on dist-test-slave-7lbf"
server_key: "0b4cb74614f2b6b54bcce02a9110cc19"
server_key_iv: "4d92180fbdcee35fdd369fa66d133381"
server_key_version: "encryptionkey@0"
I20260805 22:52:13.083436 31329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260805 22:52:13.087697 31351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:13.089417 31329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260805 22:52:13.089681 31329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "638fd766a87a4c7a837cded76c9f23c9"
format_stamp: "Formatted at 2026-08-05 22:52:13 on dist-test-slave-7lbf"
server_key: "0b4cb74614f2b6b54bcce02a9110cc19"
server_key_iv: "4d92180fbdcee35fdd369fa66d133381"
server_key_version: "encryptionkey@0"
I20260805 22:52:13.089956 31329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:13.109205 31329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:13.110255 31329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:13.110613 31329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:13.130525 31329 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:44007
I20260805 22:52:13.130584 31425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:44007 every 8 connection(s)
I20260805 22:52:13.132033 31329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:13.136197 31426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:13.138496 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 31329
I20260805 22:52:13.138711 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:13.139173 4208 external_mini_cluster.cc:1428] Setting key 21669d6c3ed89c9f61e6ca00bb3ae633
I20260805 22:52:13.149255 31426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9: Bootstrap starting.
I20260805 22:52:13.153738 31426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:13.155359 31426 log.cc:824] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:13.155686 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:13.141382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.4.28.62:44007)
0805 22:52:13.142623 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:13.142637 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:13.142691 (+ 54us) server_negotiation.cc:407] Connection header received
0805 22:52:13.144642 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:13.144673 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:13.145240 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:13.145786 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:13.146483 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:13.147896 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:13.149472 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:13.150632 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:13.150863 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:13.152279 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:13.152292 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:13.152314 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:13.153110 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:13.153256 (+ 146us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:13.153289 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:13.153423 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:13.153740 (+ 317us) server_negotiation.cc:299] Negotiation successful
0805 22:52:13.154156 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":398,"threads_started":1}
I20260805 22:52:13.159170 31426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9: No bootstrap required, opened a new log
I20260805 22:52:13.167683 31426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638fd766a87a4c7a837cded76c9f23c9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44007 } }
I20260805 22:52:13.168130 31426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:13.168212 31426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638fd766a87a4c7a837cded76c9f23c9, State: Initialized, Role: FOLLOWER
I20260805 22:52:13.168825 31426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638fd766a87a4c7a837cded76c9f23c9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44007 } }
I20260805 22:52:13.169063 31426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:13.169170 31426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:13.169296 31426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:13.171629 31426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638fd766a87a4c7a837cded76c9f23c9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44007 } }
I20260805 22:52:13.172195 31426 leader_election.cc:302] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 638fd766a87a4c7a837cded76c9f23c9; no voters:
I20260805 22:52:13.172832 31426 leader_election.cc:288] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:13.173099 31432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:13.173843 31432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [term 1 LEADER]: Becoming Leader. State: Replica: 638fd766a87a4c7a837cded76c9f23c9, State: Running, Role: LEADER
I20260805 22:52:13.174418 31432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638fd766a87a4c7a837cded76c9f23c9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44007 } }
I20260805 22:52:13.175226 31426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:13.177388 31440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 638fd766a87a4c7a837cded76c9f23c9. Latest consensus state: current_term: 1 leader_uuid: "638fd766a87a4c7a837cded76c9f23c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638fd766a87a4c7a837cded76c9f23c9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44007 } } }
I20260805 22:52:13.177897 31440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:13.177716 31433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "638fd766a87a4c7a837cded76c9f23c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638fd766a87a4c7a837cded76c9f23c9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44007 } } }
I20260805 22:52:13.178088 31433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:13.179754 31447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:13.185911 31447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:13.196055 31447 catalog_manager.cc:1347] Generated new cluster ID: 5556d40617f84d2a8e432324c5f80e30
I20260805 22:52:13.196197 31447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:13.207873 31447 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:13.209578 31447 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:13.220255 31447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9: Generated new TSK 0
I20260805 22:52:13.221380 31447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:13.235299 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44007
--builtin_ntp_servers=127.4.28.20:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 22:52:13.610589 31461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:13.610925 31461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:13.611064 31461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:13.619127 31461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:13.619306 31461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:13.619468 31461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:52:13.629223 31461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:13.630717 31461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:13.632611 31461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:13.646235 31471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:13.647037 31472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:13.647773 31475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:13.648811 31461 server_base.cc:1034] running on GCE node
I20260805 22:52:13.649554 31461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:13.650682 31461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:13.651904 31461 hybrid_clock.cc:648] HybridClock initialized: now 1785970333651832 us; error 67 us; skew 500 ppm
I20260805 22:52:13.652263 31461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:13.655131 31461 webserver.cc:460] Webserver started at http://127.4.28.1:36717/ using document root <none> and password file <none>
I20260805 22:52:13.655964 31461 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:13.656080 31461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:13.656448 31461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:13.659756 31461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "3f3d12190c3c408280de196bce3cbc8c"
format_stamp: "Formatted at 2026-08-05 22:52:13 on dist-test-slave-7lbf"
server_key: "fb6708e1ed704a552af00de1ee08b673"
server_key_iv: "8879a31d637b9b3fc9d38a1cc6568beb"
server_key_version: "encryptionkey@0"
I20260805 22:52:13.660681 31461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "3f3d12190c3c408280de196bce3cbc8c"
format_stamp: "Formatted at 2026-08-05 22:52:13 on dist-test-slave-7lbf"
server_key: "fb6708e1ed704a552af00de1ee08b673"
server_key_iv: "8879a31d637b9b3fc9d38a1cc6568beb"
server_key_version: "encryptionkey@0"
I20260805 22:52:13.666733 31461 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260805 22:52:13.670686 31486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:13.672183 31461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:52:13.672363 31461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "3f3d12190c3c408280de196bce3cbc8c"
format_stamp: "Formatted at 2026-08-05 22:52:13 on dist-test-slave-7lbf"
server_key: "fb6708e1ed704a552af00de1ee08b673"
server_key_iv: "8879a31d637b9b3fc9d38a1cc6568beb"
server_key_version: "encryptionkey@0"
I20260805 22:52:13.672602 31461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:13.690985 31461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:13.691941 31461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:13.692239 31461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:13.693418 31461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:13.695256 31461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:13.695370 31461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:13.695515 31461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:13.695590 31461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:13.729809 31461 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:35627
I20260805 22:52:13.729893 31638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:35627 every 8 connection(s)
I20260805 22:52:13.731302 31461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:52:13.731935 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 31461
I20260805 22:52:13.732115 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:52:13.732715 4208 external_mini_cluster.cc:1428] Setting key d14d22cbc75a607f00da27cbc4229c59
I20260805 22:52:13.736258 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44007
--builtin_ntp_servers=127.4.28.20:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 22:52:13.747150 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:13.734146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:43463 (local address 127.4.28.62:44007)
0805 22:52:13.734670 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:13.734677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:13.736325 (+ 1648us) server_negotiation.cc:407] Connection header received
0805 22:52:13.739020 (+ 2695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:13.739027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:13.739118 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:13.739222 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:13.740890 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:13.741737 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:13.744019 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:13.744448 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:13.744577 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:13.746324 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:13.746330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:13.746334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:13.746356 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:13.746400 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:13.746412 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:13.746485 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:13.746764 (+ 279us) server_negotiation.cc:299] Negotiation successful
0805 22:52:13.746866 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":258,"threads_started":1}
I20260805 22:52:13.748595 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:13.734880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44007 (local address 127.4.28.1:43463)
0805 22:52:13.736114 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:13.736181 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:13.738613 (+ 2432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:13.739563 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:13.739578 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:13.739717 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:13.740650 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:13.740681 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:13.741917 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:13.741922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:13.743836 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:13.743857 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:13.744716 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:13.744723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:13.745034 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:13.745909 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:52:13.745913 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:13.745973 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:52:13.746088 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:13.746511 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:13.746523 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:13.746532 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 22:52:13.746742 (+ 210us) client_negotiation.cc:240] Negotiation successful
0805 22:52:13.747201 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":347,"threads_started":1}
I20260805 22:52:13.750417 31639 heartbeater.cc:347] Connected to a master server at 127.4.28.62:44007
I20260805 22:52:13.750759 31639 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:13.751611 31639 heartbeater.cc:511] Master 127.4.28.62:44007 requested a full tablet report, sending...
I20260805 22:52:13.754204 31376 ts_manager.cc:194] Registered new tserver with Master: 3f3d12190c3c408280de196bce3cbc8c (127.4.28.1:35627)
I20260805 22:52:13.755865 31376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.1:43463
W20260805 22:52:14.105435 31644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:14.105832 31644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:14.105973 31644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:14.113667 31644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:14.113801 31644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:14.113936 31644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:52:14.123664 31644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:14.125082 31644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:14.127126 31644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:14.141567 31652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:14.141799 31656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:14.141978 31650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:14.142562 31644 server_base.cc:1034] running on GCE node
I20260805 22:52:14.143352 31644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:14.144721 31644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:14.145977 31644 hybrid_clock.cc:648] HybridClock initialized: now 1785970334145922 us; error 43 us; skew 500 ppm
I20260805 22:52:14.146380 31644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:14.149195 31644 webserver.cc:460] Webserver started at http://127.4.28.2:40633/ using document root <none> and password file <none>
I20260805 22:52:14.150007 31644 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:14.150122 31644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:14.150426 31644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:14.153558 31644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "190cc800fa5d4e189f91b2b6d4420651"
format_stamp: "Formatted at 2026-08-05 22:52:14 on dist-test-slave-7lbf"
server_key: "80da1d1496abde768d15fc460235d0a0"
server_key_iv: "20e6a51abc453239c2f72aa687bcc633"
server_key_version: "encryptionkey@0"
I20260805 22:52:14.154399 31644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "190cc800fa5d4e189f91b2b6d4420651"
format_stamp: "Formatted at 2026-08-05 22:52:14 on dist-test-slave-7lbf"
server_key: "80da1d1496abde768d15fc460235d0a0"
server_key_iv: "20e6a51abc453239c2f72aa687bcc633"
server_key_version: "encryptionkey@0"
I20260805 22:52:14.160074 31644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260805 22:52:14.164054 31663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:14.165802 31644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:52:14.165999 31644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "190cc800fa5d4e189f91b2b6d4420651"
format_stamp: "Formatted at 2026-08-05 22:52:14 on dist-test-slave-7lbf"
server_key: "80da1d1496abde768d15fc460235d0a0"
server_key_iv: "20e6a51abc453239c2f72aa687bcc633"
server_key_version: "encryptionkey@0"
I20260805 22:52:14.166249 31644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:14.198855 31644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:14.199838 31644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:14.200138 31644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:14.201283 31644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:14.203081 31644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:14.203189 31644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:14.203313 31644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:14.203394 31644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:14.237296 31644 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:36991
I20260805 22:52:14.237360 31823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:36991 every 8 connection(s)
I20260805 22:52:14.238761 31644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:52:14.246521 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 31644
I20260805 22:52:14.246730 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:52:14.247308 4208 external_mini_cluster.cc:1428] Setting key aaf0373ebc81f45ca73fd66c281ffa8a
I20260805 22:52:14.250669 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44007
--builtin_ntp_servers=127.4.28.20:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 22:52:14.253253 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.241718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:48581 (local address 127.4.28.62:44007)
0805 22:52:14.241901 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:14.241906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:14.243503 (+ 1597us) server_negotiation.cc:407] Connection header received
0805 22:52:14.245453 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:14.245459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:14.245593 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:14.245698 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:14.247297 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.248247 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.250201 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.250839 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.250998 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.252437 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:14.252444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:14.252449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:14.252474 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:14.252514 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:14.252529 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:14.252618 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:14.252826 (+ 208us) server_negotiation.cc:299] Negotiation successful
0805 22:52:14.252931 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 22:52:14.254441 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.242288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44007 (local address 127.4.28.2:48581)
0805 22:52:14.243309 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:14.243358 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:14.245097 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:14.245940 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:14.245949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:14.246098 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:14.247072 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.247105 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.248445 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.248451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.250021 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.250038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.251101 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.251107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.251338 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.252059 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:52:14.252062 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:14.252113 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:52:14.252210 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:14.252627 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:14.252634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:14.252641 (+ 7us) client_negotiation.cc:769] Sending connection context
0805 22:52:14.252794 (+ 153us) client_negotiation.cc:240] Negotiation successful
0805 22:52:14.253166 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":315,"threads_started":1}
I20260805 22:52:14.256114 31825 heartbeater.cc:347] Connected to a master server at 127.4.28.62:44007
I20260805 22:52:14.256445 31825 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:14.257297 31825 heartbeater.cc:511] Master 127.4.28.62:44007 requested a full tablet report, sending...
I20260805 22:52:14.259039 31376 ts_manager.cc:194] Registered new tserver with Master: 190cc800fa5d4e189f91b2b6d4420651 (127.4.28.2:36991)
I20260805 22:52:14.260120 31376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.2:48581
W20260805 22:52:14.619215 31829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:14.619555 31829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:14.619686 31829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:14.627255 31829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:14.627389 31829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:14.627535 31829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:52:14.637018 31829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:14.638484 31829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:14.640527 31829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:14.654994 31835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:14.654955 31836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:14.654918 31840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:14.655881 31829 server_base.cc:1034] running on GCE node
I20260805 22:52:14.656629 31829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:14.657749 31829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:14.658973 31829 hybrid_clock.cc:648] HybridClock initialized: now 1785970334658914 us; error 50 us; skew 500 ppm
I20260805 22:52:14.659322 31829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:14.664664 31829 webserver.cc:460] Webserver started at http://127.4.28.3:38549/ using document root <none> and password file <none>
I20260805 22:52:14.665464 31829 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:14.665633 31829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:14.666028 31829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:14.669203 31829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "e2da1321bdde434095716f375eb0535a"
format_stamp: "Formatted at 2026-08-05 22:52:14 on dist-test-slave-7lbf"
server_key: "860b1e26d5b9801e863cde5ce7ea8eac"
server_key_iv: "c2d0d112845a491da31fb7544cefd5e9"
server_key_version: "encryptionkey@0"
I20260805 22:52:14.670356 31829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "e2da1321bdde434095716f375eb0535a"
format_stamp: "Formatted at 2026-08-05 22:52:14 on dist-test-slave-7lbf"
server_key: "860b1e26d5b9801e863cde5ce7ea8eac"
server_key_iv: "c2d0d112845a491da31fb7544cefd5e9"
server_key_version: "encryptionkey@0"
I20260805 22:52:14.676349 31829 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260805 22:52:14.680418 31848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:14.682076 31829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:52:14.682255 31829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "e2da1321bdde434095716f375eb0535a"
format_stamp: "Formatted at 2026-08-05 22:52:14 on dist-test-slave-7lbf"
server_key: "860b1e26d5b9801e863cde5ce7ea8eac"
server_key_iv: "c2d0d112845a491da31fb7544cefd5e9"
server_key_version: "encryptionkey@0"
I20260805 22:52:14.682484 31829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:14.723768 31829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:14.724723 31829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:14.725026 31829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:14.726146 31829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:14.727828 31829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:14.727924 31829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:14.728039 31829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:14.728114 31829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:14.760385 31639 heartbeater.cc:503] Master 127.4.28.62:44007 was elected leader, sending a full tablet report...
I20260805 22:52:14.761994 31829 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:46331
I20260805 22:52:14.762017 32000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:46331 every 8 connection(s)
I20260805 22:52:14.763618 31829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:52:14.769502 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 31829
I20260805 22:52:14.769750 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:52:14.770190 4208 external_mini_cluster.cc:1428] Setting key ac21340cff93aa34ac16f476cdc0a486
I20260805 22:52:14.778558 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.766424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:59441 (local address 127.4.28.62:44007)
0805 22:52:14.766888 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:14.766893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:14.768243 (+ 1350us) server_negotiation.cc:407] Connection header received
0805 22:52:14.770354 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:14.770360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:14.770460 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:14.770600 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:14.772239 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.773401 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.775200 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.775724 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.775848 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.777659 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:14.777665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:14.777669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:14.777693 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:14.777738 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:14.777751 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:14.777834 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:14.778146 (+ 312us) server_negotiation.cc:299] Negotiation successful
0805 22:52:14.778248 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
I20260805 22:52:14.779913 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.767028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:44007 (local address 127.4.28.3:59441)
0805 22:52:14.768060 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:14.768103 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:14.770005 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:14.771076 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:14.771090 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:14.771242 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:14.772036 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.772059 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.773624 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.773631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.775007 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.775037 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.776067 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.776073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.776379 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.777162 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:52:14.777165 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:14.777216 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:52:14.777352 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:14.777881 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:14.777894 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:14.777902 (+ 8us) client_negotiation.cc:769] Sending connection context
0805 22:52:14.778126 (+ 224us) client_negotiation.cc:240] Negotiation successful
0805 22:52:14.778545 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":322,"threads_started":1}
I20260805 22:52:14.781579 32003 heartbeater.cc:347] Connected to a master server at 127.4.28.62:44007
I20260805 22:52:14.781893 32003 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:14.782747 32003 heartbeater.cc:511] Master 127.4.28.62:44007 requested a full tablet report, sending...
I20260805 22:52:14.784590 31376 ts_manager.cc:194] Registered new tserver with Master: e2da1321bdde434095716f375eb0535a (127.4.28.3:46331)
I20260805 22:52:14.785606 31376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.3:59441
I20260805 22:52:14.786859 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:52:14.799624 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.793194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.4.28.62:44007)
0805 22:52:14.793424 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:14.793431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:14.793887 (+ 456us) server_negotiation.cc:407] Connection header received
0805 22:52:14.794103 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:14.794109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:14.794202 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:14.794291 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:14.794842 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.795578 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.797097 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.797552 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.797689 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.798849 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:14.798854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:14.798858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:14.798878 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:14.798921 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:14.798933 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:14.799063 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:14.799279 (+ 216us) server_negotiation.cc:299] Negotiation successful
0805 22:52:14.799343 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 22:52:14.804128 31372 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:52:14.811426 31372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:52:14.838687 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.824620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:46331 (local address 127.0.0.1:54426)
0805 22:52:14.827203 (+ 2583us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:14.827223 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:14.827434 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:14.828562 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:14.828571 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:14.828621 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:14.829150 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.829162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.831286 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.831291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.835077 (+ 3786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.835093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.837666 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.837671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.837937 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.837986 (+ 49us) client_negotiation.cc:769] Sending connection context
0805 22:52:14.838108 (+ 122us) client_negotiation.cc:240] Negotiation successful
0805 22:52:14.838311 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2455,"thread_start_us":988,"threads_started":1}
I20260805 22:52:14.838713 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.825340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.4.28.3:46331)
0805 22:52:14.826360 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:14.826369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:14.827325 (+ 956us) server_negotiation.cc:407] Connection header received
0805 22:52:14.827668 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:14.827677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:14.827955 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:14.828140 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:14.829376 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.831075 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.835329 (+ 4254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.836983 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.837110 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.837435 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:14.838199 (+ 764us) server_negotiation.cc:299] Negotiation successful
0805 22:52:14.838369 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":1243,"threads_started":1}
I20260805 22:52:14.843509 31911 tablet_service.cc:1467] Processing CreateTablet for tablet de2bb3990664419e81c919824221e95e (DEFAULT_TABLE table=test-table [id=f3e799b543514e4f9ed9aa374d9bf9a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:14.844995 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.824749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.4.28.1:35627)
0805 22:52:14.836835 (+ 12086us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:14.836845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:14.836883 (+ 38us) server_negotiation.cc:407] Connection header received
0805 22:52:14.836993 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:14.837002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:14.837287 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:14.838305 (+ 1018us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:14.838588 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.840096 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.842136 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.843629 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.843832 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.844119 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:14.844405 (+ 286us) server_negotiation.cc:299] Negotiation successful
0805 22:52:14.844593 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11872,"thread_start_us":12313,"threads_started":1}
I20260805 22:52:14.845350 31911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2bb3990664419e81c919824221e95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:14.845943 32022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.826478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:36991 (local address 127.0.0.1:38812)
0805 22:52:14.836834 (+ 10356us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:14.836854 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:14.836997 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:14.838583 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:14.838588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:14.838616 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:14.839431 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.839447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.841562 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.841567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.843074 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.843089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.845182 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.845189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.845355 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.845386 (+ 31us) client_negotiation.cc:769] Sending connection context
0805 22:52:14.845473 (+ 87us) client_negotiation.cc:240] Negotiation successful
0805 22:52:14.845635 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10218,"thread_start_us":579,"threads_started":1}
I20260805 22:52:14.846300 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.824484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35627 (local address 127.0.0.1:40544)
0805 22:52:14.825414 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:14.825459 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:14.825700 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:14.837579 (+ 11879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:14.837585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:14.837615 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:14.838079 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.838093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.840250 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.840255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.841895 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:14.841912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.843891 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.843899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:14.844084 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.844122 (+ 38us) client_negotiation.cc:769] Sending connection context
0805 22:52:14.844217 (+ 95us) client_negotiation.cc:240] Negotiation successful
0805 22:52:14.844361 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":5,"thread_start_us":12804,"threads_started":1}
I20260805 22:52:14.846644 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:14.826885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.4.28.2:36991)
0805 22:52:14.837958 (+ 11073us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:14.837966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:14.837997 (+ 31us) server_negotiation.cc:407] Connection header received
0805 22:52:14.838103 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:14.838110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:14.838357 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:14.839647 (+ 1290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:14.839890 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.841357 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.843271 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:14.844655 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:14.844792 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:14.845038 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:14.845704 (+ 666us) server_negotiation.cc:299] Negotiation successful
0805 22:52:14.845897 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10896,"thread_start_us":10765,"threads_started":1}
I20260805 22:52:14.850414 31550 tablet_service.cc:1467] Processing CreateTablet for tablet de2bb3990664419e81c919824221e95e (DEFAULT_TABLE table=test-table [id=f3e799b543514e4f9ed9aa374d9bf9a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:14.850414 31735 tablet_service.cc:1467] Processing CreateTablet for tablet de2bb3990664419e81c919824221e95e (DEFAULT_TABLE table=test-table [id=f3e799b543514e4f9ed9aa374d9bf9a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:14.852110 31735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2bb3990664419e81c919824221e95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:14.852504 31550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2bb3990664419e81c919824221e95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:14.858489 32029 tablet_bootstrap.cc:492] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: Bootstrap starting.
I20260805 22:52:14.862286 32029 tablet_bootstrap.cc:654] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:14.863421 32029 log.cc:824] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:14.866844 32029 tablet_bootstrap.cc:492] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: No bootstrap required, opened a new log
I20260805 22:52:14.867143 32029 ts_tablet_manager.cc:1397] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260805 22:52:14.868065 32031 tablet_bootstrap.cc:492] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: Bootstrap starting.
I20260805 22:52:14.870287 32032 tablet_bootstrap.cc:492] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: Bootstrap starting.
I20260805 22:52:14.873477 32032 tablet_bootstrap.cc:654] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:14.873631 32031 tablet_bootstrap.cc:654] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:14.874714 32032 log.cc:824] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:14.874902 32031 log.cc:824] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:14.878338 32031 tablet_bootstrap.cc:492] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: No bootstrap required, opened a new log
I20260805 22:52:14.878368 32032 tablet_bootstrap.cc:492] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: No bootstrap required, opened a new log
I20260805 22:52:14.878659 32031 ts_tablet_manager.cc:1397] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260805 22:52:14.878665 32032 ts_tablet_manager.cc:1397] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260805 22:52:14.878513 32029 raft_consensus.cc:348] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } }
I20260805 22:52:14.879063 32029 raft_consensus.cc:374] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:14.879184 32029 raft_consensus.cc:729] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2da1321bdde434095716f375eb0535a, State: Initialized, Role: FOLLOWER
I20260805 22:52:14.880048 32029 consensus_queue.cc:260] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } }
I20260805 22:52:14.882407 32003 heartbeater.cc:503] Master 127.4.28.62:44007 was elected leader, sending a full tablet report...
I20260805 22:52:14.883186 32029 ts_tablet_manager.cc:1428] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260805 22:52:14.888177 32031 raft_consensus.cc:348] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } }
I20260805 22:52:14.888695 32031 raft_consensus.cc:374] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:14.888819 32031 raft_consensus.cc:729] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190cc800fa5d4e189f91b2b6d4420651, State: Initialized, Role: FOLLOWER
I20260805 22:52:14.889561 32031 consensus_queue.cc:260] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } }
I20260805 22:52:14.889433 32032 raft_consensus.cc:348] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } }
I20260805 22:52:14.889964 32032 raft_consensus.cc:374] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:14.890077 32032 raft_consensus.cc:729] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3d12190c3c408280de196bce3cbc8c, State: Initialized, Role: FOLLOWER
I20260805 22:52:14.890861 32032 consensus_queue.cc:260] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } }
I20260805 22:52:14.891038 31825 heartbeater.cc:503] Master 127.4.28.62:44007 was elected leader, sending a full tablet report...
I20260805 22:52:14.891778 32031 ts_tablet_manager.cc:1428] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260805 22:52:14.893939 32032 ts_tablet_manager.cc:1428] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260805 22:52:14.984119 31640 tablet.cc:2369] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:52:14.991019 31827 tablet.cc:2369] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:52:15.015434 32004 tablet.cc:2369] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:52:15.024608 32035 raft_consensus.cc:482] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:52:15.024928 32035 raft_consensus.cc:504] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } }
I20260805 22:52:15.026785 32035 leader_election.cc:288] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f3d12190c3c408280de196bce3cbc8c (127.4.28.1:35627), 190cc800fa5d4e189f91b2b6d4420651 (127.4.28.2:36991)
I20260805 22:52:15.032557 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:15.027384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:52627 (local address 127.4.28.1:35627)
0805 22:52:15.027593 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:15.027598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:15.027619 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:52:15.027698 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:15.027704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:15.027799 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:15.027899 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:15.028387 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.029114 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.030894 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.031687 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.031810 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:15.031914 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:15.032183 (+ 269us) server_negotiation.cc:299] Negotiation successful
0805 22:52:15.032316 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260805 22:52:15.032595 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:15.027102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:35627 (local address 127.4.28.3:52627)
0805 22:52:15.027323 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:15.027345 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:15.027515 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:15.027981 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:15.027987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:15.028022 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:15.028202 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:15.028212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.029254 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.029258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:15.030715 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:15.030728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.031852 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.031859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:15.032017 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:15.032043 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 22:52:15.032127 (+ 84us) client_negotiation.cc:240] Negotiation successful
0805 22:52:15.032273 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":1}
I20260805 22:52:15.033229 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:15.027191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:36991 (local address 127.4.28.3:40007)
0805 22:52:15.028134 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:15.028151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:15.028280 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:15.028609 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:15.028613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:15.028637 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:15.028946 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:15.028956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.030011 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.030015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:15.031401 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:15.031415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.032551 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.032557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:15.032745 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:15.032782 (+ 37us) client_negotiation.cc:769] Sending connection context
0805 22:52:15.032863 (+ 81us) client_negotiation.cc:240] Negotiation successful
0805 22:52:15.032990 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":578,"threads_started":1}
I20260805 22:52:15.033315 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:15.027570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:40007 (local address 127.4.28.2:36991)
0805 22:52:15.027797 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:15.027803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:15.028254 (+ 451us) server_negotiation.cc:407] Connection header received
0805 22:52:15.028415 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:15.028419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:15.028493 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:15.028568 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:15.029114 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.029860 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.031602 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.032365 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.032483 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:15.032572 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:15.032913 (+ 341us) server_negotiation.cc:299] Negotiation successful
0805 22:52:15.033070 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260805 22:52:15.033881 31578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2bb3990664419e81c919824221e95e" candidate_uuid: "e2da1321bdde434095716f375eb0535a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3d12190c3c408280de196bce3cbc8c" is_pre_election: true
I20260805 22:52:15.034375 31761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2bb3990664419e81c919824221e95e" candidate_uuid: "e2da1321bdde434095716f375eb0535a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190cc800fa5d4e189f91b2b6d4420651" is_pre_election: true
I20260805 22:52:15.034587 31578 raft_consensus.cc:2454] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2da1321bdde434095716f375eb0535a in term 0.
I20260805 22:52:15.034970 31761 raft_consensus.cc:2454] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2da1321bdde434095716f375eb0535a in term 0.
I20260805 22:52:15.035383 31851 leader_election.cc:302] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3d12190c3c408280de196bce3cbc8c, e2da1321bdde434095716f375eb0535a; no voters:
I20260805 22:52:15.035863 32035 raft_consensus.cc:2781] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:52:15.036033 32035 raft_consensus.cc:482] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:52:15.036144 32035 raft_consensus.cc:3037] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:15.038542 32035 raft_consensus.cc:504] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } }
I20260805 22:52:15.039227 32035 leader_election.cc:288] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [CANDIDATE]: Term 1 election: Requested vote from peers 3f3d12190c3c408280de196bce3cbc8c (127.4.28.1:35627), 190cc800fa5d4e189f91b2b6d4420651 (127.4.28.2:36991)
I20260805 22:52:15.039955 31578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2bb3990664419e81c919824221e95e" candidate_uuid: "e2da1321bdde434095716f375eb0535a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3d12190c3c408280de196bce3cbc8c"
I20260805 22:52:15.039954 31761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2bb3990664419e81c919824221e95e" candidate_uuid: "e2da1321bdde434095716f375eb0535a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190cc800fa5d4e189f91b2b6d4420651"
I20260805 22:52:15.040217 31578 raft_consensus.cc:3037] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:15.040217 31761 raft_consensus.cc:3037] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:15.043808 31761 raft_consensus.cc:2454] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2da1321bdde434095716f375eb0535a in term 1.
I20260805 22:52:15.043812 31578 raft_consensus.cc:2454] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2da1321bdde434095716f375eb0535a in term 1.
I20260805 22:52:15.044484 31850 leader_election.cc:302] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 190cc800fa5d4e189f91b2b6d4420651, 3f3d12190c3c408280de196bce3cbc8c, e2da1321bdde434095716f375eb0535a; no voters:
I20260805 22:52:15.044891 32035 raft_consensus.cc:2781] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:15.045421 32035 raft_consensus.cc:686] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 1 LEADER]: Becoming Leader. State: Replica: e2da1321bdde434095716f375eb0535a, State: Running, Role: LEADER
I20260805 22:52:15.045990 32035 consensus_queue.cc:237] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } }
I20260805 22:52:15.051313 31371 catalog_manager.cc:5495] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a reported cstate change: term changed from 0 to 1, leader changed from <none> to e2da1321bdde434095716f375eb0535a (127.4.28.3). New cstate: current_term: 1 leader_uuid: "e2da1321bdde434095716f375eb0535a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2da1321bdde434095716f375eb0535a" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 46331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 } health_report { overall_health: UNKNOWN } } }
I20260805 22:52:15.083740 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:15.078199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.4.28.3:46331)
0805 22:52:15.078440 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:15.078447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:15.078485 (+ 38us) server_negotiation.cc:407] Connection header received
0805 22:52:15.078733 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:15.078738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:15.078824 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:15.078980 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:52:15.079370 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.080152 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.081627 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.082060 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.082160 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:15.082647 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:52:15.083202 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:52:15.083348 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:15.083425 (+ 77us) server_negotiation.cc:299] Negotiation successful
0805 22:52:15.083466 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260805 22:52:15.092536 31578 raft_consensus.cc:1261] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [term 1 FOLLOWER]: Refusing update from remote peer e2da1321bdde434095716f375eb0535a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:15.092543 31761 raft_consensus.cc:1261] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [term 1 FOLLOWER]: Refusing update from remote peer e2da1321bdde434095716f375eb0535a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:15.093593 32040 consensus_queue.cc:1035] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [LEADER]: Connected to new peer: Peer: permanent_uuid: "190cc800fa5d4e189f91b2b6d4420651" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:15.093991 32035 consensus_queue.cc:1035] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3d12190c3c408280de196bce3cbc8c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:15.104789 32046 mvcc.cc:205] Tried to move back new op lower bound from 7315334492528652288 to 7315334492352659456. Current Snapshot: MvccSnapshot[applied={T|T < 7315334492528652288}]
I20260805 22:52:15.106045 32048 mvcc.cc:205] Tried to move back new op lower bound from 7315334492528652288 to 7315334492352659456. Current Snapshot: MvccSnapshot[applied={T|T < 7315334492528652288}]
I20260805 22:52:15.122443 31376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:15.122723 31376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:15.126706 31376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 638fd766a87a4c7a837cded76c9f23c9: Sending DeleteTablet for 3 replicas of tablet de2bb3990664419e81c919824221e95e
I20260805 22:52:15.128183 31550 tablet_service.cc:1514] Processing DeleteTablet for tablet de2bb3990664419e81c919824221e95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:15 UTC) from {username='slave'} at 127.0.0.1:40544
I20260805 22:52:15.128223 31735 tablet_service.cc:1514] Processing DeleteTablet for tablet de2bb3990664419e81c919824221e95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:15 UTC) from {username='slave'} at 127.0.0.1:38812
I20260805 22:52:15.128374 31911 tablet_service.cc:1514] Processing DeleteTablet for tablet de2bb3990664419e81c919824221e95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:15 UTC) from {username='slave'} at 127.0.0.1:54426
I20260805 22:52:15.130404 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 31461
I20260805 22:52:15.140643 32062 tablet_replica.cc:330] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: stopping tablet replica
I20260805 22:52:15.141223 32062 raft_consensus.cc:2229] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:15.141856 32062 raft_consensus.cc:2258] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:15.142757 32061 tablet_replica.cc:330] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: stopping tablet replica
I20260805 22:52:15.143457 32061 raft_consensus.cc:2229] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:15.144117 32061 raft_consensus.cc:2258] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:15.146890 32061 ts_tablet_manager.cc:1905] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:15.154460 32061 ts_tablet_manager.cc:1918] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:15.154793 32061 log.cc:1196] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/de2bb3990664419e81c919824221e95e
I20260805 22:52:15.155128 32062 ts_tablet_manager.cc:1905] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:15.155646 32061 ts_tablet_manager.cc:1939] T de2bb3990664419e81c919824221e95e P 190cc800fa5d4e189f91b2b6d4420651: Deleting consensus metadata
I20260805 22:52:15.157820 31353 catalog_manager.cc:4841] TS 190cc800fa5d4e189f91b2b6d4420651 (127.4.28.2:36991): tablet de2bb3990664419e81c919824221e95e (table test-table [id=f3e799b543514e4f9ed9aa374d9bf9a2]) successfully deleted
I20260805 22:52:15.160161 32064 tablet_replica.cc:330] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: stopping tablet replica
I20260805 22:52:15.160759 32064 raft_consensus.cc:2229] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:52:15.161609 32064 raft_consensus.cc:2258] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:15.161881 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:15.131978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.4.28.1:35627)
0805 22:52:15.132184 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:15.132192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:15.147466 (+ 15274us) server_negotiation.cc:407] Connection header received
0805 22:52:15.148185 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:15.148191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:15.148310 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:15.148395 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:15.149106 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.150303 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.152440 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.153163 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.153282 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:15.159941 (+ 6659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:15.159955 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:15.159975 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:15.160820 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:15.160897 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:15.160931 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:15.161071 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:15.161312 (+ 241us) server_negotiation.cc:299] Negotiation successful
0805 22:52:15.161464 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260805 22:52:15.163067 32062 ts_tablet_manager.cc:1918] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:15.163312 32062 log.cc:1196] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/de2bb3990664419e81c919824221e95e
I20260805 22:52:15.163487 31634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:15.164137 32064 ts_tablet_manager.cc:1905] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:15.172828 32064 ts_tablet_manager.cc:1918] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:15.173153 32064 log.cc:1196] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/de2bb3990664419e81c919824221e95e
I20260805 22:52:15.173826 32064 ts_tablet_manager.cc:1939] T de2bb3990664419e81c919824221e95e P e2da1321bdde434095716f375eb0535a: Deleting consensus metadata
I20260805 22:52:15.176017 31353 catalog_manager.cc:4841] TS e2da1321bdde434095716f375eb0535a (127.4.28.3:46331): tablet de2bb3990664419e81c919824221e95e (table test-table [id=f3e799b543514e4f9ed9aa374d9bf9a2]) successfully deleted
I20260805 22:52:15.267118 32062 ts_tablet_manager.cc:1939] T de2bb3990664419e81c919824221e95e P 3f3d12190c3c408280de196bce3cbc8c: Deleting consensus metadata
I20260805 22:52:15.269207 31354 catalog_manager.cc:4841] TS 3f3d12190c3c408280de196bce3cbc8c (127.4.28.1:35627): tablet de2bb3990664419e81c919824221e95e (table test-table [id=f3e799b543514e4f9ed9aa374d9bf9a2]) successfully deleted
I20260805 22:52:15.269544 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 31461
I20260805 22:52:15.295487 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 31644
I20260805 22:52:15.304524 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:15.296435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.4.28.2:36991)
0805 22:52:15.296679 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:15.296686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:15.296711 (+ 25us) server_negotiation.cc:407] Connection header received
0805 22:52:15.296839 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:15.296845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:15.297003 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:15.297130 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:15.297551 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.298818 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.300523 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.301204 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.301350 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:15.302667 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:15.302680 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:15.302702 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:15.303567 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:15.303639 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:15.303668 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:15.303812 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:15.303995 (+ 183us) server_negotiation.cc:299] Negotiation successful
0805 22:52:15.304129 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260805 22:52:15.305437 31817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:15.396528 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 31644
I20260805 22:52:15.432274 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 31829
I20260805 22:52:15.440898 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:15.433106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.4.28.3:46331)
0805 22:52:15.433313 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:15.433320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:15.433345 (+ 25us) server_negotiation.cc:407] Connection header received
0805 22:52:15.433499 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:15.433506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:15.433659 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:15.433784 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:15.434182 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.435378 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.436768 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:15.437475 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:15.437790 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:15.438897 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:15.438912 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:15.438934 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:15.439897 (+ 963us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:15.439975 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:15.440009 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:15.440141 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:15.440335 (+ 194us) server_negotiation.cc:299] Negotiation successful
0805 22:52:15.440462 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260805 22:52:15.441844 31996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:15.566473 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 31829
I20260805 22:52:15.591828 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 31329
I20260805 22:52:15.593093 31420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:15.705570 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 31329
2026-08-05T22:52:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3160 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:15 dist-test-slave-7lbf krb5kdc[32073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:15 dist-test-slave-7lbf krb5kdc[32073](info): set up 2 sockets
Aug 05 22:52:15 dist-test-slave-7lbf krb5kdc[32073](info): commencing operation
krb5kdc: starting...
W20260805 22:52:16.950937 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.004s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:17 dist-test-slave-7lbf krb5kdc[32073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:52:17.315282 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:40503
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:17.690006 32093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:17.690351 32093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:17.690502 32093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:17.698110 32093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:17.698262 32093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:17.698340 32093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:17.698418 32093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:17.698503 32093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:17.708972 32093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:40503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:40503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:17.710574 32093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:17.712145 32093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:17.725148 32100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:17.725116 32101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:17.725077 32106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:17.726447 32093 server_base.cc:1034] running on GCE node
I20260805 22:52:17.727330 32093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:17.728953 32093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:17.730192 32093 hybrid_clock.cc:648] HybridClock initialized: now 1785970337730134 us; error 50 us; skew 500 ppm
I20260805 22:52:17.730618 32093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:17 dist-test-slave-7lbf krb5kdc[32073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970337, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:17.734028 32093 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:52:17.735847 32093 webserver.cc:460] Webserver started at http://127.4.28.62:39819/ using document root <none> and password file <none>
I20260805 22:52:17.736641 32093 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:17.736778 32093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:17.737196 32093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:17.740581 32093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "799e73dedd1a44c5a347dd56ec67af00"
format_stamp: "Formatted at 2026-08-05 22:52:17 on dist-test-slave-7lbf"
server_key: "3919dc764d600ccf1e7b3b0aa6882c2d"
server_key_iv: "1027498c73fbf4b8085b4453da39d3c0"
server_key_version: "encryptionkey@0"
I20260805 22:52:17.741441 32093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "799e73dedd1a44c5a347dd56ec67af00"
format_stamp: "Formatted at 2026-08-05 22:52:17 on dist-test-slave-7lbf"
server_key: "3919dc764d600ccf1e7b3b0aa6882c2d"
server_key_iv: "1027498c73fbf4b8085b4453da39d3c0"
server_key_version: "encryptionkey@0"
I20260805 22:52:17.747306 32093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260805 22:52:17.751307 32114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:17.752820 32093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 22:52:17.753015 32093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "799e73dedd1a44c5a347dd56ec67af00"
format_stamp: "Formatted at 2026-08-05 22:52:17 on dist-test-slave-7lbf"
server_key: "3919dc764d600ccf1e7b3b0aa6882c2d"
server_key_iv: "1027498c73fbf4b8085b4453da39d3c0"
server_key_version: "encryptionkey@0"
I20260805 22:52:17.753229 32093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:17.773285 32093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:17.777714 32093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:17.778059 32093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:17.798069 32093 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:40503
I20260805 22:52:17.798105 32195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:40503 every 8 connection(s)
I20260805 22:52:17.799556 32093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:17.801818 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 32093
I20260805 22:52:17.802047 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:17.802611 4208 external_mini_cluster.cc:1428] Setting key 1333f65c674a26e5345111208ca20607
I20260805 22:52:17.804589 32196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:52:17 dist-test-slave-7lbf krb5kdc[32073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:17.819557 32196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00: Bootstrap starting.
I20260805 22:52:17.823496 32196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:17.824949 32196 log.cc:824] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:17.828996 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:17.804656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.4.28.62:40503)
0805 22:52:17.805510 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:17.805561 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:17.805612 (+ 51us) server_negotiation.cc:407] Connection header received
0805 22:52:17.807344 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:17.807371 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:17.807831 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:17.808208 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:17.810184 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:17.811691 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:17.813619 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:17.814376 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:17.814554 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:17.820032 (+ 5478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:17.820042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:17.820061 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:17.820115 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:17.823309 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:17.824413 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:17.824423 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:17.824431 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:17.824671 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:17.825232 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:17.825238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:17.825241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:17.826037 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:17.826298 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:17.827052 (+ 754us) server_negotiation.cc:299] Negotiation successful
0805 22:52:17.827546 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":362,"threads_started":1}
I20260805 22:52:17.829018 32196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00: No bootstrap required, opened a new log
I20260805 22:52:17.838002 32196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e73dedd1a44c5a347dd56ec67af00" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40503 } }
I20260805 22:52:17.838550 32196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:17.838687 32196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 799e73dedd1a44c5a347dd56ec67af00, State: Initialized, Role: FOLLOWER
I20260805 22:52:17.839308 32196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e73dedd1a44c5a347dd56ec67af00" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40503 } }
I20260805 22:52:17.839566 32196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:17.839700 32196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:17.839867 32196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:17.842465 32196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e73dedd1a44c5a347dd56ec67af00" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40503 } }
I20260805 22:52:17.843186 32196 leader_election.cc:302] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 799e73dedd1a44c5a347dd56ec67af00; no voters:
I20260805 22:52:17.843849 32196 leader_election.cc:288] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:17.844012 32206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:17.844714 32206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [term 1 LEADER]: Becoming Leader. State: Replica: 799e73dedd1a44c5a347dd56ec67af00, State: Running, Role: LEADER
I20260805 22:52:17.845286 32206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e73dedd1a44c5a347dd56ec67af00" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40503 } }
I20260805 22:52:17.846112 32196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:17.848441 32207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "799e73dedd1a44c5a347dd56ec67af00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e73dedd1a44c5a347dd56ec67af00" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40503 } } }
I20260805 22:52:17.848399 32208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 799e73dedd1a44c5a347dd56ec67af00. Latest consensus state: current_term: 1 leader_uuid: "799e73dedd1a44c5a347dd56ec67af00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799e73dedd1a44c5a347dd56ec67af00" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40503 } } }
I20260805 22:52:17.848909 32207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:17.848963 32208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:17.851166 32213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:17.857095 32213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:17.868650 32213 catalog_manager.cc:1347] Generated new cluster ID: e77c3c0d1de24e5e93b9c3d40f4adc93
I20260805 22:52:17.868793 32213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:17.897122 32213 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:17.898586 32213 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:17.904790 32213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00: Generated new TSK 0
I20260805 22:52:17.905771 32213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.28.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
Entry for principal kudu/127.4.28.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab.
I20260805 22:52:18.014257 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40503
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:18.389822 32231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:18.390210 32231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:18.390362 32231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:18.398149 32231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:18.398286 32231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:18.398430 32231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:52:18.409253 32231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.1.keytab
--principal=kudu/127.4.28.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:18.410787 32231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:18.412310 32231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:18.426731 32236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:18.426781 32239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:18.426805 32237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:18.427667 32231 server_base.cc:1034] running on GCE node
I20260805 22:52:18.428363 32231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:18.429447 32231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:18.430696 32231 hybrid_clock.cc:648] HybridClock initialized: now 1785970338430630 us; error 58 us; skew 500 ppm
I20260805 22:52:18.431032 32231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:18 dist-test-slave-7lbf krb5kdc[32073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970338, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:18.434433 32231 init.cc:377] Logged in from keytab as kudu/127.4.28.1@KRBTEST.COM (short username kudu)
I20260805 22:52:18.436098 32231 webserver.cc:460] Webserver started at http://127.4.28.1:39147/ using document root <none> and password file <none>
I20260805 22:52:18.436882 32231 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:18.437017 32231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:18.437363 32231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:18.440714 32231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "3af471b2995341e7b1d28d50a7681b7c"
format_stamp: "Formatted at 2026-08-05 22:52:18 on dist-test-slave-7lbf"
server_key: "80ffb122a8e8827bec30954ea408674c"
server_key_iv: "365c6b7dca31fc04d8069c7e337fa827"
server_key_version: "encryptionkey@0"
I20260805 22:52:18.441713 32231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "3af471b2995341e7b1d28d50a7681b7c"
format_stamp: "Formatted at 2026-08-05 22:52:18 on dist-test-slave-7lbf"
server_key: "80ffb122a8e8827bec30954ea408674c"
server_key_iv: "365c6b7dca31fc04d8069c7e337fa827"
server_key_version: "encryptionkey@0"
I20260805 22:52:18.448113 32231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260805 22:52:18.452350 32249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:18.454075 32231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:52:18.454273 32231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "3af471b2995341e7b1d28d50a7681b7c"
format_stamp: "Formatted at 2026-08-05 22:52:18 on dist-test-slave-7lbf"
server_key: "80ffb122a8e8827bec30954ea408674c"
server_key_iv: "365c6b7dca31fc04d8069c7e337fa827"
server_key_version: "encryptionkey@0"
I20260805 22:52:18.454499 32231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:18.468451 32231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:18.472954 32231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:18.473263 32231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:18.474462 32231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:18.476014 32231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:18.476099 32231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:18.476225 32231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:18.476277 32231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:18.509399 32231 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:42761
I20260805 22:52:18.509459 32402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:42761 every 8 connection(s)
I20260805 22:52:18.510849 32231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:52:18.511364 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 32231
I20260805 22:52:18.511554 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:52:18.512183 4208 external_mini_cluster.cc:1428] Setting key aad59b0882c2a851c61abf648e224d66
Aug 05 22:52:18 dist-test-slave-7lbf krb5kdc[32073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970338, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.1@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:18.533345 32405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:18.513916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:54253 (local address 127.4.28.62:40503)
0805 22:52:18.514390 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:18.514396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:18.515650 (+ 1254us) server_negotiation.cc:407] Connection header received
0805 22:52:18.517628 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:18.517634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:18.517726 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:18.517824 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:18.519541 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:18.520268 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:18.522191 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:18.522629 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:18.522751 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:18.526984 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:18.526990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:18.526994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:18.527039 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:18.529765 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:18.530679 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:18.530686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:18.530689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:18.530874 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:18.531375 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:18.531380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:18.531382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:18.531739 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:18.531895 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:18.532723 (+ 828us) server_negotiation.cc:299] Negotiation successful
0805 22:52:18.532978 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":245,"threads_started":1}
I20260805 22:52:18.534287 32406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:18.514608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40503 (local address 127.4.28.1:54253)
0805 22:52:18.515481 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:18.515537 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:18.517296 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:18.518119 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:18.518129 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:18.518698 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:18.519333 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:18.519353 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:18.520442 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:18.520446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:18.522024 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:18.522041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:18.522888 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:18.522896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:18.523143 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:18.523836 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:52:18.523862 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:18.526635 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:18.529974 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:18.529985 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:18.530002 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:18.530518 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:18.531028 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:18.531031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:18.531033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:18.531248 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:18.531880 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:18.531888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:18.532316 (+ 428us) client_negotiation.cc:769] Sending connection context
0805 22:52:18.532578 (+ 262us) client_negotiation.cc:240] Negotiation successful
0805 22:52:18.533041 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":383,"threads_started":1}
I20260805 22:52:18.535976 32403 heartbeater.cc:347] Connected to a master server at 127.4.28.62:40503
I20260805 22:52:18.536329 32403 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:18.537184 32403 heartbeater.cc:511] Master 127.4.28.62:40503 requested a full tablet report, sending...
I20260805 22:52:18.539688 32138 ts_manager.cc:194] Registered new tserver with Master: 3af471b2995341e7b1d28d50a7681b7c (127.4.28.1:42761)
I20260805 22:52:18.541276 32138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.1@KRBTEST.COM'} at 127.4.28.1:54253
WARNING: no policy specified for kudu/127.4.28.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
Entry for principal kudu/127.4.28.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab.
I20260805 22:52:18.597677 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40503
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:18.978886 32410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:18.979213 32410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:18.979355 32410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:18.987023 32410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:18.987166 32410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:18.987321 32410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:52:18.997828 32410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.2.keytab
--principal=kudu/127.4.28.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:18.999270 32410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:19.000711 32410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:19.015141 32418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:19.015098 32420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:19.015182 32422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:19.016182 32410 server_base.cc:1034] running on GCE node
I20260805 22:52:19.016844 32410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:19.017979 32410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:19.019212 32410 hybrid_clock.cc:648] HybridClock initialized: now 1785970339019158 us; error 46 us; skew 500 ppm
I20260805 22:52:19.019574 32410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:19 dist-test-slave-7lbf krb5kdc[32073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:19.023329 32410 init.cc:377] Logged in from keytab as kudu/127.4.28.2@KRBTEST.COM (short username kudu)
I20260805 22:52:19.025130 32410 webserver.cc:460] Webserver started at http://127.4.28.2:43327/ using document root <none> and password file <none>
I20260805 22:52:19.025923 32410 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:19.026052 32410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:19.026335 32410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:19.029470 32410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "e8223e5586a2463e9b06d6b9b2d6024d"
format_stamp: "Formatted at 2026-08-05 22:52:19 on dist-test-slave-7lbf"
server_key: "ce7d07b5001d50ec62cd76a16dafb809"
server_key_iv: "e0d4dcd72cae894ce4bf4b67bda7e4f2"
server_key_version: "encryptionkey@0"
I20260805 22:52:19.030370 32410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "e8223e5586a2463e9b06d6b9b2d6024d"
format_stamp: "Formatted at 2026-08-05 22:52:19 on dist-test-slave-7lbf"
server_key: "ce7d07b5001d50ec62cd76a16dafb809"
server_key_iv: "e0d4dcd72cae894ce4bf4b67bda7e4f2"
server_key_version: "encryptionkey@0"
I20260805 22:52:19.036109 32410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260805 22:52:19.039913 32432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:19.041409 32410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:52:19.041649 32410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "e8223e5586a2463e9b06d6b9b2d6024d"
format_stamp: "Formatted at 2026-08-05 22:52:19 on dist-test-slave-7lbf"
server_key: "ce7d07b5001d50ec62cd76a16dafb809"
server_key_iv: "e0d4dcd72cae894ce4bf4b67bda7e4f2"
server_key_version: "encryptionkey@0"
I20260805 22:52:19.041870 32410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:19.061780 32410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:19.066433 32410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:19.066743 32410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:19.067977 32410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:19.069960 32410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:19.070075 32410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:19.070222 32410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:19.070290 32410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:19.103303 32410 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:45235
I20260805 22:52:19.103350 32606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:45235 every 8 connection(s)
I20260805 22:52:19.104729 32410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:52:19.114626 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 32410
I20260805 22:52:19.114811 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:52:19.115273 4208 external_mini_cluster.cc:1428] Setting key e4572d9f2a377ac648e75c8b47859223
Aug 05 22:52:19 dist-test-slave-7lbf krb5kdc[32073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.2@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:19.131507 32611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.107603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:57191 (local address 127.4.28.62:40503)
0805 22:52:19.108093 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:19.108098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:19.109225 (+ 1127us) server_negotiation.cc:407] Connection header received
0805 22:52:19.111240 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:19.111246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:19.111342 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:19.111445 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:19.113241 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.114090 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.115858 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.116394 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.116545 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.124960 (+ 8415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:19.124967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:19.124970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:19.125019 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:19.127871 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:19.128840 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:19.128846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:19.128849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:19.128996 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:19.129497 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:19.129501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:19.129504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:19.129851 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:19.130021 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:19.130842 (+ 821us) server_negotiation.cc:299] Negotiation successful
0805 22:52:19.131109 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260805 22:52:19.132614 32612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.108214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40503 (local address 127.4.28.2:57191)
0805 22:52:19.109031 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:19.109078 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:19.110899 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:19.111738 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:19.111748 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:19.112338 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:19.112995 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.113017 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.114264 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.114268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.115684 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.115700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.116642 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.116648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.116883 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.121908 (+ 5025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:52:19.121953 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:19.124595 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:19.128102 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:19.128112 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:19.128130 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:19.128670 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:19.129131 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:19.129135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:19.129138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:19.129366 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:19.130006 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:19.130015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:19.130419 (+ 404us) client_negotiation.cc:769] Sending connection context
0805 22:52:19.130722 (+ 303us) client_negotiation.cc:240] Negotiation successful
0805 22:52:19.131350 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":299,"threads_started":1}
I20260805 22:52:19.134496 32608 heartbeater.cc:347] Connected to a master server at 127.4.28.62:40503
I20260805 22:52:19.134883 32608 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:19.135723 32608 heartbeater.cc:511] Master 127.4.28.62:40503 requested a full tablet report, sending...
I20260805 22:52:19.137616 32138 ts_manager.cc:194] Registered new tserver with Master: e8223e5586a2463e9b06d6b9b2d6024d (127.4.28.2:45235)
I20260805 22:52:19.138520 32138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.2@KRBTEST.COM'} at 127.4.28.2:57191
WARNING: no policy specified for kudu/127.4.28.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
Entry for principal kudu/127.4.28.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab.
I20260805 22:52:19.192937 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40503
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:52:19.544750 32403 heartbeater.cc:503] Master 127.4.28.62:40503 was elected leader, sending a full tablet report...
W20260805 22:52:19.568363 32615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:19.568722 32615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:19.568872 32615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:19.576535 32615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:19.576709 32615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:19.576897 32615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:52:19.587592 32615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.3.keytab
--principal=kudu/127.4.28.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:40503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:19.589212 32615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:19.590721 32615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:19.605084 32622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:19.605111 32621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:19.606045 32625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:19.606817 32615 server_base.cc:1034] running on GCE node
I20260805 22:52:19.607543 32615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:19.608657 32615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:19.609934 32615 hybrid_clock.cc:648] HybridClock initialized: now 1785970339609805 us; error 81 us; skew 500 ppm
I20260805 22:52:19.610301 32615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:19 dist-test-slave-7lbf krb5kdc[32073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:19.614285 32615 init.cc:377] Logged in from keytab as kudu/127.4.28.3@KRBTEST.COM (short username kudu)
I20260805 22:52:19.619810 32615 webserver.cc:460] Webserver started at http://127.4.28.3:46105/ using document root <none> and password file <none>
I20260805 22:52:19.620776 32615 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:19.621011 32615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:19.621388 32615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:19.625370 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "645638baada24a71884f1a3b0c711fe5"
format_stamp: "Formatted at 2026-08-05 22:52:19 on dist-test-slave-7lbf"
server_key: "c6b9b37c36799654329099c7e9f2ab37"
server_key_iv: "9dcfb4f8ca0ded7390ad324c091b8de9"
server_key_version: "encryptionkey@0"
I20260805 22:52:19.626345 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "645638baada24a71884f1a3b0c711fe5"
format_stamp: "Formatted at 2026-08-05 22:52:19 on dist-test-slave-7lbf"
server_key: "c6b9b37c36799654329099c7e9f2ab37"
server_key_iv: "9dcfb4f8ca0ded7390ad324c091b8de9"
server_key_version: "encryptionkey@0"
I20260805 22:52:19.632637 32615 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260805 22:52:19.636968 32636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:19.638815 32615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260805 22:52:19.639034 32615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "645638baada24a71884f1a3b0c711fe5"
format_stamp: "Formatted at 2026-08-05 22:52:19 on dist-test-slave-7lbf"
server_key: "c6b9b37c36799654329099c7e9f2ab37"
server_key_iv: "9dcfb4f8ca0ded7390ad324c091b8de9"
server_key_version: "encryptionkey@0"
I20260805 22:52:19.639261 32615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:19.660561 32615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:19.668742 32615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:19.669117 32615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:19.670361 32615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:19.671962 32615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:19.672051 32615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:19.672183 32615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:19.672238 32615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:19.706595 32615 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:35187
I20260805 22:52:19.706634 336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:35187 every 8 connection(s)
I20260805 22:52:19.707929 32615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:52:19.711439 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 32615
I20260805 22:52:19.711650 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:52:19.712261 4208 external_mini_cluster.cc:1428] Setting key ec9399561c53bc7e18bab3edc3d8811d
Aug 05 22:52:19 dist-test-slave-7lbf krb5kdc[32073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.3@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:19.729367 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.710662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:42109 (local address 127.4.28.62:40503)
0805 22:52:19.711179 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:19.711184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:19.712279 (+ 1095us) server_negotiation.cc:407] Connection header received
0805 22:52:19.714654 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:19.714658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:19.714728 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:19.714813 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:19.716666 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.717495 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.719124 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.719550 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.719652 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.723310 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:19.723316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:19.723320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:19.723359 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:19.725903 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:19.726794 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:19.726800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:19.726804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:19.726963 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:19.727461 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:19.727466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:19.727469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:19.727873 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:19.728073 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:19.728863 (+ 790us) server_negotiation.cc:299] Negotiation successful
0805 22:52:19.729075 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":283,"threads_started":1}
I20260805 22:52:19.730324 341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.711229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:40503 (local address 127.4.28.3:42109)
0805 22:52:19.712102 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:19.712144 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:19.714323 (+ 2179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:19.715130 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:19.715143 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:19.715852 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:19.716493 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.716513 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.717659 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.717663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.718969 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.718984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.719806 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.719812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.720032 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.720691 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:52:19.720720 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:19.723067 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:19.726076 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:19.726084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:19.726101 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:19.726656 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:19.727108 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:52:19.727112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:52:19.727114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:52:19.727328 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:52:19.728027 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:19.728035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:19.728405 (+ 370us) client_negotiation.cc:769] Sending connection context
0805 22:52:19.728672 (+ 267us) client_negotiation.cc:240] Negotiation successful
0805 22:52:19.729122 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":372,"threads_started":1}
I20260805 22:52:19.731827 338 heartbeater.cc:347] Connected to a master server at 127.4.28.62:40503
I20260805 22:52:19.732157 338 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:19.732934 338 heartbeater.cc:511] Master 127.4.28.62:40503 requested a full tablet report, sending...
I20260805 22:52:19.734514 32138 ts_manager.cc:194] Registered new tserver with Master: 645638baada24a71884f1a3b0c711fe5 (127.4.28.3:35187)
I20260805 22:52:19.735414 32138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.28.3@KRBTEST.COM'} at 127.4.28.3:42109
I20260805 22:52:19.740661 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:52:19.759665 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.745211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.4.28.62:40503)
0805 22:52:19.745552 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:19.745558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:19.745873 (+ 315us) server_negotiation.cc:407] Connection header received
0805 22:52:19.746006 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:19.746010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:19.746084 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:19.746183 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:19.747827 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.748672 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.750279 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.750749 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.750872 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.753492 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:19.753498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:19.753501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:19.753603 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:19.756450 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:19.757305 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:19.757311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:19.757314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:19.757479 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:19.757960 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:19.757966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:19.757968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:19.758391 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:19.758687 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:19.759070 (+ 383us) server_negotiation.cc:299] Negotiation successful
0805 22:52:19.759332 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 22:52:19.764118 32138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:52:19.771534 32138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:52:19.798240 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.785198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:45235 (local address 127.0.0.1:35114)
0805 22:52:19.786149 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:19.786190 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:19.786354 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:19.787428 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:19.787438 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:19.787475 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:19.787929 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.787943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.790645 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.790652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.793657 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.793701 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.796555 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.796563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.796860 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.796924 (+ 64us) client_negotiation.cc:769] Sending connection context
0805 22:52:19.797503 (+ 579us) client_negotiation.cc:240] Negotiation successful
0805 22:52:19.797819 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":487,"threads_started":1}
I20260805 22:52:19.798233 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.785416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.4.28.2:45235)
0805 22:52:19.786117 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:19.786126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:19.786285 (+ 159us) server_negotiation.cc:407] Connection header received
0805 22:52:19.786583 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:19.786590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:19.786873 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:19.787098 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:19.788179 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.790398 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.794023 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.795847 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.796077 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.796348 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:19.797736 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0805 22:52:19.797924 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":395,"threads_started":1}
I20260805 22:52:19.803004 32504 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6f8a91c02741ab985911422da4aa3a (DEFAULT_TABLE table=test-table [id=dffb35c269ec4bc9b399a4f564ef7503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:19.805029 32504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6f8a91c02741ab985911422da4aa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:19.807430 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.786574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42761 (local address 127.0.0.1:33410)
0805 22:52:19.787135 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:19.787152 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:19.787275 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:19.799044 (+ 11769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:19.799049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:19.799081 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:19.799418 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.799429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.801415 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.801421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.803749 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.803769 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.806481 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.806488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.806713 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.806752 (+ 39us) client_negotiation.cc:769] Sending connection context
0805 22:52:19.806849 (+ 97us) client_negotiation.cc:240] Negotiation successful
0805 22:52:19.807002 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":268,"threads_started":1}
I20260805 22:52:19.808535 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.786916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.4.28.1:42761)
0805 22:52:19.797715 (+ 10799us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:19.797725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:19.797759 (+ 34us) server_negotiation.cc:407] Connection header received
0805 22:52:19.797863 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:19.797872 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:19.798184 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:19.798778 (+ 594us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:19.799625 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.801230 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.804161 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.805885 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.806037 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.806302 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:19.807980 (+ 1678us) server_negotiation.cc:299] Negotiation successful
0805 22:52:19.808185 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10568,"thread_start_us":10382,"threads_started":1}
I20260805 22:52:19.813158 32312 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6f8a91c02741ab985911422da4aa3a (DEFAULT_TABLE table=test-table [id=dffb35c269ec4bc9b399a4f564ef7503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:19.815065 32312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6f8a91c02741ab985911422da4aa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:19.816119 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.787191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:35187 (local address 127.0.0.1:35238)
0805 22:52:19.795717 (+ 8526us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:19.795743 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:19.795921 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:19.806906 (+ 10985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:19.806912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:19.806949 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:19.807407 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.807427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.810926 (+ 3499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.810932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.813113 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.813136 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.815250 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.815258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.815443 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.815495 (+ 52us) client_negotiation.cc:769] Sending connection context
0805 22:52:19.815577 (+ 82us) client_negotiation.cc:240] Negotiation successful
0805 22:52:19.815743 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8347,"thread_start_us":8838,"threads_started":1}
I20260805 22:52:19.817487 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.787708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.4.28.3:35187)
0805 22:52:19.805940 (+ 18232us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:19.805949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:19.805988 (+ 39us) server_negotiation.cc:407] Connection header received
0805 22:52:19.806103 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:19.806110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:19.806395 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:19.807687 (+ 1292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:19.808007 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.810722 (+ 2715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.813410 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.814947 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.816594 (+ 1647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.816857 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:19.816947 (+ 90us) server_negotiation.cc:299] Negotiation successful
0805 22:52:19.817125 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18013,"thread_start_us":18880,"threads_started":1}
I20260805 22:52:19.821540 32709 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6f8a91c02741ab985911422da4aa3a (DEFAULT_TABLE table=test-table [id=dffb35c269ec4bc9b399a4f564ef7503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:19.822868 363 tablet_bootstrap.cc:492] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: Bootstrap starting.
I20260805 22:52:19.823760 32709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6f8a91c02741ab985911422da4aa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:19.826964 363 tablet_bootstrap.cc:654] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:19.828061 363 log.cc:824] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:19.831219 363 tablet_bootstrap.cc:492] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: No bootstrap required, opened a new log
I20260805 22:52:19.831534 363 ts_tablet_manager.cc:1397] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260805 22:52:19.832640 366 tablet_bootstrap.cc:492] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: Bootstrap starting.
I20260805 22:52:19.836575 366 tablet_bootstrap.cc:654] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:19.838156 366 log.cc:824] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:19.839090 367 tablet_bootstrap.cc:492] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: Bootstrap starting.
I20260805 22:52:19.842417 363 raft_consensus.cc:348] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } }
I20260805 22:52:19.842782 367 tablet_bootstrap.cc:654] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:19.842945 363 raft_consensus.cc:374] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:19.843084 363 raft_consensus.cc:729] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8223e5586a2463e9b06d6b9b2d6024d, State: Initialized, Role: FOLLOWER
I20260805 22:52:19.844105 367 log.cc:824] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:19.843926 363 consensus_queue.cc:260] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } }
I20260805 22:52:19.845243 366 tablet_bootstrap.cc:492] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: No bootstrap required, opened a new log
I20260805 22:52:19.845727 366 ts_tablet_manager.cc:1397] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260805 22:52:19.855468 363 ts_tablet_manager.cc:1428] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: Time spent starting tablet: real 0.024s user 0.014s sys 0.011s
I20260805 22:52:19.856086 32608 heartbeater.cc:503] Master 127.4.28.62:40503 was elected leader, sending a full tablet report...
W20260805 22:52:19.856863 32610 tablet.cc:2369] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:52:19.858312 366 raft_consensus.cc:348] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } }
I20260805 22:52:19.858736 366 raft_consensus.cc:374] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:19.858837 366 raft_consensus.cc:729] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af471b2995341e7b1d28d50a7681b7c, State: Initialized, Role: FOLLOWER
I20260805 22:52:19.859519 366 consensus_queue.cc:260] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } }
I20260805 22:52:19.859956 367 tablet_bootstrap.cc:492] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: No bootstrap required, opened a new log
I20260805 22:52:19.860296 367 ts_tablet_manager.cc:1397] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.009s
I20260805 22:52:19.861580 366 ts_tablet_manager.cc:1428] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20260805 22:52:19.869642 367 raft_consensus.cc:348] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } }
I20260805 22:52:19.870080 367 raft_consensus.cc:374] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:19.870162 367 raft_consensus.cc:729] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645638baada24a71884f1a3b0c711fe5, State: Initialized, Role: FOLLOWER
I20260805 22:52:19.870798 367 consensus_queue.cc:260] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } }
I20260805 22:52:19.871902 338 heartbeater.cc:503] Master 127.4.28.62:40503 was elected leader, sending a full tablet report...
I20260805 22:52:19.872438 367 ts_tablet_manager.cc:1428] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260805 22:52:19.901439 374 raft_consensus.cc:482] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:52:19.901834 374 raft_consensus.cc:504] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } }
I20260805 22:52:19.903599 374 leader_election.cc:288] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8223e5586a2463e9b06d6b9b2d6024d (127.4.28.2:45235), 3af471b2995341e7b1d28d50a7681b7c (127.4.28.1:42761)
I20260805 22:52:19.909461 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.904019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:51109 (local address 127.4.28.2:45235)
0805 22:52:19.904212 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:19.904217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:19.904326 (+ 109us) server_negotiation.cc:407] Connection header received
0805 22:52:19.904565 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:19.904569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:19.904634 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:19.904729 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:19.905208 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.905996 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.907735 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.908535 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.908665 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.908767 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:19.909075 (+ 308us) server_negotiation.cc:299] Negotiation successful
0805 22:52:19.909213 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 22:52:19.909461 341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.904003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:45235 (local address 127.4.28.3:51109)
0805 22:52:19.904226 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:19.904248 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:19.904410 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:19.904784 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:19.904789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:19.904822 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:19.905046 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.905065 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.906149 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.906155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.907559 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.907576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.908711 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.908718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.908907 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.908934 (+ 27us) client_negotiation.cc:769] Sending connection context
0805 22:52:19.909020 (+ 86us) client_negotiation.cc:240] Negotiation successful
0805 22:52:19.909161 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":42}
I20260805 22:52:19.910553 32533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6f8a91c02741ab985911422da4aa3a" candidate_uuid: "645638baada24a71884f1a3b0c711fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" is_pre_election: true
I20260805 22:52:19.910807 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.904792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:40775 (local address 127.4.28.1:42761)
0805 22:52:19.905067 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:19.905073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:19.905494 (+ 421us) server_negotiation.cc:407] Connection header received
0805 22:52:19.905702 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:19.905706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:19.905770 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:19.905848 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:19.906456 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.907176 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.908936 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.909884 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.910022 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.910126 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:19.910417 (+ 291us) server_negotiation.cc:299] Negotiation successful
0805 22:52:19.910565 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260805 22:52:19.910913 375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.904477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:42761 (local address 127.4.28.3:40775)
0805 22:52:19.905382 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:19.905398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:19.905558 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:19.905904 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:19.905908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:19.905936 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:19.906290 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.906300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.907345 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.907350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.908755 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:19.908769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.910101 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.910107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:19.910250 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.910275 (+ 25us) client_negotiation.cc:769] Sending connection context
0805 22:52:19.910360 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 22:52:19.910494 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":3,"thread_start_us":619,"threads_started":1}
I20260805 22:52:19.911145 32533 raft_consensus.cc:2454] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645638baada24a71884f1a3b0c711fe5 in term 0.
I20260805 22:52:19.911859 32640 leader_election.cc:302] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 645638baada24a71884f1a3b0c711fe5, e8223e5586a2463e9b06d6b9b2d6024d; no voters:
I20260805 22:52:19.912264 374 raft_consensus.cc:2781] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:52:19.912122 32343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6f8a91c02741ab985911422da4aa3a" candidate_uuid: "645638baada24a71884f1a3b0c711fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af471b2995341e7b1d28d50a7681b7c" is_pre_election: true
I20260805 22:52:19.912395 374 raft_consensus.cc:482] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:52:19.912500 374 raft_consensus.cc:3037] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:19.912639 32343 raft_consensus.cc:2454] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645638baada24a71884f1a3b0c711fe5 in term 0.
I20260805 22:52:19.915149 374 raft_consensus.cc:504] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } }
I20260805 22:52:19.916067 374 leader_election.cc:288] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [CANDIDATE]: Term 1 election: Requested vote from peers e8223e5586a2463e9b06d6b9b2d6024d (127.4.28.2:45235), 3af471b2995341e7b1d28d50a7681b7c (127.4.28.1:42761)
I20260805 22:52:19.916689 32343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6f8a91c02741ab985911422da4aa3a" candidate_uuid: "645638baada24a71884f1a3b0c711fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af471b2995341e7b1d28d50a7681b7c"
I20260805 22:52:19.916664 32533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6f8a91c02741ab985911422da4aa3a" candidate_uuid: "645638baada24a71884f1a3b0c711fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8223e5586a2463e9b06d6b9b2d6024d"
I20260805 22:52:19.916953 32343 raft_consensus.cc:3037] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:19.917025 32533 raft_consensus.cc:3037] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:19.920568 32533 raft_consensus.cc:2454] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645638baada24a71884f1a3b0c711fe5 in term 1.
I20260805 22:52:19.920557 32343 raft_consensus.cc:2454] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645638baada24a71884f1a3b0c711fe5 in term 1.
I20260805 22:52:19.921221 32640 leader_election.cc:302] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 645638baada24a71884f1a3b0c711fe5, e8223e5586a2463e9b06d6b9b2d6024d; no voters:
I20260805 22:52:19.921615 374 raft_consensus.cc:2781] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:19.922158 374 raft_consensus.cc:686] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 1 LEADER]: Becoming Leader. State: Replica: 645638baada24a71884f1a3b0c711fe5, State: Running, Role: LEADER
I20260805 22:52:19.922775 374 consensus_queue.cc:237] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } }
I20260805 22:52:19.927759 32137 catalog_manager.cc:5495] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 645638baada24a71884f1a3b0c711fe5 (127.4.28.3). New cstate: current_term: 1 leader_uuid: "645638baada24a71884f1a3b0c711fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "645638baada24a71884f1a3b0c711fe5" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 35187 } health_report { overall_health: HEALTHY } } }
I20260805 22:52:19.954792 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:19.948796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.4.28.3:35187)
0805 22:52:19.949014 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:19.949019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:19.949039 (+ 20us) server_negotiation.cc:407] Connection header received
0805 22:52:19.949187 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:19.949191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:19.949278 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:19.949427 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:52:19.950040 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.950820 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.952358 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:19.952882 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:19.953004 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:19.953616 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:52:19.954253 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:52:19.954385 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:19.954486 (+ 101us) server_negotiation.cc:299] Negotiation successful
0805 22:52:19.954575 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260805 22:52:19.959669 339 tablet.cc:2369] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:52:19.964545 32343 raft_consensus.cc:1261] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [term 1 FOLLOWER]: Refusing update from remote peer 645638baada24a71884f1a3b0c711fe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:19.964529 32533 raft_consensus.cc:1261] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [term 1 FOLLOWER]: Refusing update from remote peer 645638baada24a71884f1a3b0c711fe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:19.965873 376 consensus_queue.cc:1035] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af471b2995341e7b1d28d50a7681b7c" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 42761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:19.966410 374 consensus_queue.cc:1035] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8223e5586a2463e9b06d6b9b2d6024d" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 45235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:19.976509 379 mvcc.cc:205] Tried to move back new op lower bound from 7315334512482402304 to 7315334512328130560. Current Snapshot: MvccSnapshot[applied={T|T < 7315334512482402304}]
I20260805 22:52:19.995198 32138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:19.995494 32138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:19.999378 32138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 799e73dedd1a44c5a347dd56ec67af00: Sending DeleteTablet for 3 replicas of tablet 5e6f8a91c02741ab985911422da4aa3a
I20260805 22:52:20.000828 32504 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6f8a91c02741ab985911422da4aa3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:19 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:35114
I20260805 22:52:20.001132 32312 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6f8a91c02741ab985911422da4aa3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:19 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:33410
I20260805 22:52:20.001264 32709 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6f8a91c02741ab985911422da4aa3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:19 UTC) from {username='kudu', principal='kudu/127.4.28.62@KRBTEST.COM'} at 127.0.0.1:35238
I20260805 22:52:20.003829 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 32231
I20260805 22:52:20.014384 399 tablet_replica.cc:330] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: stopping tablet replica
I20260805 22:52:20.015000 399 raft_consensus.cc:2229] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:52:20.015781 399 raft_consensus.cc:2258] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:20.021960 399 ts_tablet_manager.cc:1905] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:20.028280 398 tablet_replica.cc:330] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: stopping tablet replica
I20260805 22:52:20.029254 397 tablet_replica.cc:330] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: stopping tablet replica
I20260805 22:52:20.029326 398 raft_consensus.cc:2229] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:20.030056 398 raft_consensus.cc:2258] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:20.030926 397 raft_consensus.cc:2229] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:20.031517 397 raft_consensus.cc:2258] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 05 22:52:20 dist-test-slave-7lbf krb5kdc[32073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.1@KRBTEST.COM
I20260805 22:52:20.034094 398 ts_tablet_manager.cc:1905] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:20.034149 399 ts_tablet_manager.cc:1918] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:20.034350 399 log.cc:1196] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/5e6f8a91c02741ab985911422da4aa3a
I20260805 22:52:20.035043 399 ts_tablet_manager.cc:1939] T 5e6f8a91c02741ab985911422da4aa3a P 645638baada24a71884f1a3b0c711fe5: Deleting consensus metadata
I20260805 22:52:20.037329 32115 catalog_manager.cc:4841] TS 645638baada24a71884f1a3b0c711fe5 (127.4.28.3:35187): tablet 5e6f8a91c02741ab985911422da4aa3a (table test-table [id=dffb35c269ec4bc9b399a4f564ef7503]) successfully deleted
I20260805 22:52:20.041159 397 ts_tablet_manager.cc:1905] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:20.042956 398 ts_tablet_manager.cc:1918] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:20.043236 398 log.cc:1196] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-0/wal/wals/5e6f8a91c02741ab985911422da4aa3a
I20260805 22:52:20.043881 398 ts_tablet_manager.cc:1939] T 5e6f8a91c02741ab985911422da4aa3a P 3af471b2995341e7b1d28d50a7681b7c: Deleting consensus metadata
I20260805 22:52:20.044028 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:20.005211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.4.28.1:42761)
0805 22:52:20.005421 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:20.005428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:20.019422 (+ 13994us) server_negotiation.cc:407] Connection header received
0805 22:52:20.020055 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:20.020063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:20.020208 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:20.020293 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:20.022693 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:20.024036 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:20.026496 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:20.027347 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:20.027474 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:20.035484 (+ 8010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:20.035503 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:20.035524 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:20.035580 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:20.039378 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:20.040522 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:20.040536 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:20.040539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:20.040761 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:20.041389 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:20.041406 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:20.041409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:20.041977 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:20.042297 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:20.043014 (+ 717us) server_negotiation.cc:299] Negotiation successful
0805 22:52:20.043495 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:52:20.045830 32397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:20.046241 32115 catalog_manager.cc:4841] TS 3af471b2995341e7b1d28d50a7681b7c (127.4.28.1:42761): tablet 5e6f8a91c02741ab985911422da4aa3a (table test-table [id=dffb35c269ec4bc9b399a4f564ef7503]) successfully deleted
I20260805 22:52:20.050673 397 ts_tablet_manager.cc:1918] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:20.050890 397 log.cc:1196] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/5e6f8a91c02741ab985911422da4aa3a
I20260805 22:52:20.051440 397 ts_tablet_manager.cc:1939] T 5e6f8a91c02741ab985911422da4aa3a P e8223e5586a2463e9b06d6b9b2d6024d: Deleting consensus metadata
I20260805 22:52:20.053254 32115 catalog_manager.cc:4841] TS e8223e5586a2463e9b06d6b9b2d6024d (127.4.28.2:45235): tablet 5e6f8a91c02741ab985911422da4aa3a (table test-table [id=dffb35c269ec4bc9b399a4f564ef7503]) successfully deleted
I20260805 22:52:20.176323 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 32231
I20260805 22:52:20.201892 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 32410
Aug 05 22:52:20 dist-test-slave-7lbf krb5kdc[32073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.2@KRBTEST.COM
I20260805 22:52:20.220799 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:20.202908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.4.28.2:45235)
0805 22:52:20.203135 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:20.203142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:20.203166 (+ 24us) server_negotiation.cc:407] Connection header received
0805 22:52:20.203239 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:20.203244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:20.203352 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:20.203472 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:20.204759 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:20.206025 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:20.207439 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:20.208112 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:20.208257 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:20.212896 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:20.212919 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:20.212942 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:20.212998 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:20.216948 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:20.218163 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:20.218176 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:20.218180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:20.218363 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:20.218844 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:20.218849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:20.218853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:20.219402 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:20.219740 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:20.220132 (+ 392us) server_negotiation.cc:299] Negotiation successful
0805 22:52:20.220400 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:52:20.221661 32600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:20.314463 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 32410
I20260805 22:52:20.345981 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 32615
Aug 05 22:52:20 dist-test-slave-7lbf krb5kdc[32073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.3@KRBTEST.COM
I20260805 22:52:20.363605 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:20.347025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.4.28.3:35187)
0805 22:52:20.347256 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:20.347263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:20.347287 (+ 24us) server_negotiation.cc:407] Connection header received
0805 22:52:20.347352 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:20.347355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:20.347427 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:20.347535 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:20.348961 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:20.349841 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:20.351198 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:20.351681 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:20.351813 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:20.356446 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:20.356462 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:20.356482 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:20.356534 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:20.360225 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:20.361080 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:20.361089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:20.361092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:20.361261 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:20.361830 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:20.361834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:20.361836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:20.362262 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:20.362603 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:20.362938 (+ 335us) server_negotiation.cc:299] Negotiation successful
0805 22:52:20.363204 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260805 22:52:20.364374 331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:20.493646 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 32615
I20260805 22:52:20.526031 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 32093
I20260805 22:52:20.527153 32190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:20.640877 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 32093
2026-08-05T22:52:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4935 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-05T22:52:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:20Z Disabled control of system clock
I20260805 22:52:20.749365 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37917
--webserver_interface=127.4.28.62
--webserver_port=0
--builtin_ntp_servers=127.4.28.20:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:37917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 22:52:21.126502 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:21.126839 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:21.126933 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:21.134655 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:21.134784 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:21.134822 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:21.134881 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:21.134931 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:21.144704 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:37917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:37917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:21.146139 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:21.148109 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:21.162384 423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:21.162400 420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:21.162480 426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:21.163272 412 server_base.cc:1034] running on GCE node
I20260805 22:52:21.164170 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:21.165861 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:21.167126 412 hybrid_clock.cc:648] HybridClock initialized: now 1785970341167051 us; error 68 us; skew 500 ppm
I20260805 22:52:21.167534 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:21.170279 412 webserver.cc:460] Webserver started at http://127.4.28.62:43115/ using document root <none> and password file <none>
I20260805 22:52:21.171110 412 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:21.171231 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:21.171584 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:21.174937 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "8c753ee53bce4852a77e01b6fef9e23e"
format_stamp: "Formatted at 2026-08-05 22:52:21 on dist-test-slave-7lbf"
server_key: "4d2fbce6bb9e75cb8545f045d9b14331"
server_key_iv: "fa46461079f508a2615cf76ff4ff3929"
server_key_version: "encryptionkey@0"
I20260805 22:52:21.175830 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "8c753ee53bce4852a77e01b6fef9e23e"
format_stamp: "Formatted at 2026-08-05 22:52:21 on dist-test-slave-7lbf"
server_key: "4d2fbce6bb9e75cb8545f045d9b14331"
server_key_iv: "fa46461079f508a2615cf76ff4ff3929"
server_key_version: "encryptionkey@0"
I20260805 22:52:21.181710 412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260805 22:52:21.185812 435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:21.187340 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:52:21.187559 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "8c753ee53bce4852a77e01b6fef9e23e"
format_stamp: "Formatted at 2026-08-05 22:52:21 on dist-test-slave-7lbf"
server_key: "4d2fbce6bb9e75cb8545f045d9b14331"
server_key_iv: "fa46461079f508a2615cf76ff4ff3929"
server_key_version: "encryptionkey@0"
I20260805 22:52:21.187789 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:21.201630 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:21.202613 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:21.202939 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:21.221829 412 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:37917
I20260805 22:52:21.221851 507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:37917 every 8 connection(s)
I20260805 22:52:21.223341 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:21.226008 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 412
I20260805 22:52:21.226207 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:21.226786 4208 external_mini_cluster.cc:1428] Setting key 670596cc91b45fe1af6fda6ff39b691b
I20260805 22:52:21.228593 508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:21.241668 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:21.228982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.4.28.62:37917)
0805 22:52:21.230118 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:21.230131 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:21.230179 (+ 48us) server_negotiation.cc:407] Connection header received
0805 22:52:21.231912 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:21.231940 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:21.232508 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:21.232931 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:21.233590 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:21.235033 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:21.236530 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:21.237281 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:21.237477 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:21.238875 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:21.238883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:21.238906 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:21.239602 (+ 696us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:21.239695 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:21.239722 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:21.239851 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:21.240137 (+ 286us) server_negotiation.cc:299] Negotiation successful
0805 22:52:21.240504 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":399,"threads_started":1}
I20260805 22:52:21.241698 508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e: Bootstrap starting.
I20260805 22:52:21.245820 508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:21.246992 508 log.cc:824] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:21.250516 508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e: No bootstrap required, opened a new log
I20260805 22:52:21.258606 508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c753ee53bce4852a77e01b6fef9e23e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37917 } }
I20260805 22:52:21.259105 508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:21.259210 508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c753ee53bce4852a77e01b6fef9e23e, State: Initialized, Role: FOLLOWER
I20260805 22:52:21.259814 508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c753ee53bce4852a77e01b6fef9e23e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37917 } }
I20260805 22:52:21.260068 508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:21.260202 508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:21.260378 508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:21.262813 508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c753ee53bce4852a77e01b6fef9e23e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37917 } }
I20260805 22:52:21.263406 508 leader_election.cc:302] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c753ee53bce4852a77e01b6fef9e23e; no voters:
I20260805 22:52:21.264195 508 leader_election.cc:288] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:21.264289 517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:21.265058 517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [term 1 LEADER]: Becoming Leader. State: Replica: 8c753ee53bce4852a77e01b6fef9e23e, State: Running, Role: LEADER
I20260805 22:52:21.265779 517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c753ee53bce4852a77e01b6fef9e23e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37917 } }
I20260805 22:52:21.266901 508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:21.268940 520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c753ee53bce4852a77e01b6fef9e23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c753ee53bce4852a77e01b6fef9e23e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37917 } } }
I20260805 22:52:21.269063 518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c753ee53bce4852a77e01b6fef9e23e. Latest consensus state: current_term: 1 leader_uuid: "8c753ee53bce4852a77e01b6fef9e23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c753ee53bce4852a77e01b6fef9e23e" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 37917 } } }
I20260805 22:52:21.269680 520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:21.269841 518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:21.278447 531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:21.283811 531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:21.293110 531 catalog_manager.cc:1347] Generated new cluster ID: 84f99b98af4c4fa6b35fd3aaefe9c106
I20260805 22:52:21.293252 531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:21.320225 531 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:21.322319 531 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:21.337025 531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e: Generated new TSK 0
I20260805 22:52:21.338073 531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:21.347718 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.1:0
--local_ip_for_outbound_sockets=127.4.28.1
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37917
--builtin_ntp_servers=127.4.28.20:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 22:52:21.726608 548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:21.726950 548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:21.727072 548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:21.734912 548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:21.735046 548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:21.735198 548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.1
I20260805 22:52:21.745261 548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.28.1
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.1
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:21.746820 548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:21.748880 548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:21.764324 556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:21.764446 555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:21.764371 559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:21.765148 548 server_base.cc:1034] running on GCE node
I20260805 22:52:21.765892 548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:21.767129 548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:21.768393 548 hybrid_clock.cc:648] HybridClock initialized: now 1785970341768316 us; error 62 us; skew 500 ppm
I20260805 22:52:21.768774 548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:21.771559 548 webserver.cc:460] Webserver started at http://127.4.28.1:46705/ using document root <none> and password file <none>
I20260805 22:52:21.772433 548 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:21.772543 548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:21.772861 548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:21.776274 548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/data/instance:
uuid: "77276d2c6bbf436488574b5b8e631b8d"
format_stamp: "Formatted at 2026-08-05 22:52:21 on dist-test-slave-7lbf"
server_key: "4755e3ead3851a2d64944583ce557d0c"
server_key_iv: "098fd372b8bfdc5c171565e234cd85a6"
server_key_version: "encryptionkey@0"
I20260805 22:52:21.777200 548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance:
uuid: "77276d2c6bbf436488574b5b8e631b8d"
format_stamp: "Formatted at 2026-08-05 22:52:21 on dist-test-slave-7lbf"
server_key: "4755e3ead3851a2d64944583ce557d0c"
server_key_iv: "098fd372b8bfdc5c171565e234cd85a6"
server_key_version: "encryptionkey@0"
I20260805 22:52:21.783484 548 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260805 22:52:21.787927 568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:21.789659 548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:52:21.789876 548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/wal
uuid: "77276d2c6bbf436488574b5b8e631b8d"
format_stamp: "Formatted at 2026-08-05 22:52:21 on dist-test-slave-7lbf"
server_key: "4755e3ead3851a2d64944583ce557d0c"
server_key_iv: "098fd372b8bfdc5c171565e234cd85a6"
server_key_version: "encryptionkey@0"
I20260805 22:52:21.790132 548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:21.813868 548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:21.814851 548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:21.815160 548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:21.816329 548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:21.818177 548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:21.818290 548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:21.818418 548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:21.818495 548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:21.852509 548 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.1:41635
I20260805 22:52:21.852577 739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.1:41635 every 8 connection(s)
I20260805 22:52:21.854095 548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/data/info.pb
I20260805 22:52:21.854995 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 548
I20260805 22:52:21.855186 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-0/wal/instance
I20260805 22:52:21.855772 4208 external_mini_cluster.cc:1428] Setting key 6d7fc9c0f9af30074ebe6fa9e47f5726
I20260805 22:52:21.859331 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.2:0
--local_ip_for_outbound_sockets=127.4.28.2
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37917
--builtin_ntp_servers=127.4.28.20:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 22:52:21.870479 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:21.857091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:39779 (local address 127.4.28.62:37917)
0805 22:52:21.857625 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:21.857632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:21.859265 (+ 1633us) server_negotiation.cc:407] Connection header received
0805 22:52:21.862057 (+ 2792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:21.862063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:21.862147 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:21.862248 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:21.863836 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:21.864646 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:21.866954 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:21.867489 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:21.867609 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:21.869436 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:21.869443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:21.869447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:21.869472 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:21.869552 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:21.869572 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:21.869654 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:21.870044 (+ 390us) server_negotiation.cc:299] Negotiation successful
0805 22:52:21.870158 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260805 22:52:21.871906 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:21.857825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37917 (local address 127.4.28.1:39779)
0805 22:52:21.859060 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:21.859126 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:21.861673 (+ 2547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:21.862564 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:21.862578 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:21.862725 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:21.863609 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:21.863636 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:21.864848 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:21.864853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:21.866750 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:21.866769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:21.867753 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:21.867760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:21.868114 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:21.868966 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:52:21.868970 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:21.869035 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:52:21.869166 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:21.869661 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:21.869673 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:21.869682 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 22:52:21.869897 (+ 215us) client_negotiation.cc:240] Negotiation successful
0805 22:52:21.870393 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":314,"threads_started":1}
I20260805 22:52:21.873883 740 heartbeater.cc:347] Connected to a master server at 127.4.28.62:37917
I20260805 22:52:21.874280 740 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:21.875166 740 heartbeater.cc:511] Master 127.4.28.62:37917 requested a full tablet report, sending...
I20260805 22:52:21.877804 462 ts_manager.cc:194] Registered new tserver with Master: 77276d2c6bbf436488574b5b8e631b8d (127.4.28.1:41635)
I20260805 22:52:21.880007 462 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.1:39779
W20260805 22:52:22.233035 745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:22.233346 745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:22.233464 745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:22.241766 745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:22.241909 745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:22.242072 745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.2
I20260805 22:52:22.251955 745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.28.2
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.2
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:22.253376 745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:22.255576 745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:22.270437 754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:22.270464 758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:22.270517 755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:22.271911 745 server_base.cc:1034] running on GCE node
I20260805 22:52:22.272653 745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:22.273811 745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:22.275084 745 hybrid_clock.cc:648] HybridClock initialized: now 1785970342275023 us; error 44 us; skew 500 ppm
I20260805 22:52:22.275465 745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:22.278374 745 webserver.cc:460] Webserver started at http://127.4.28.2:39913/ using document root <none> and password file <none>
I20260805 22:52:22.279219 745 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:22.279342 745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:22.279708 745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:22.283090 745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/data/instance:
uuid: "eb30e33ed064429a8880af02a4ed7755"
format_stamp: "Formatted at 2026-08-05 22:52:22 on dist-test-slave-7lbf"
server_key: "151676bd8455030e61c0caa77c133b0f"
server_key_iv: "fc193cc29330929b954fb28dbc9104b5"
server_key_version: "encryptionkey@0"
I20260805 22:52:22.284124 745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance:
uuid: "eb30e33ed064429a8880af02a4ed7755"
format_stamp: "Formatted at 2026-08-05 22:52:22 on dist-test-slave-7lbf"
server_key: "151676bd8455030e61c0caa77c133b0f"
server_key_iv: "fc193cc29330929b954fb28dbc9104b5"
server_key_version: "encryptionkey@0"
I20260805 22:52:22.290400 745 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260805 22:52:22.295085 765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:22.296788 745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260805 22:52:22.297003 745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/wal
uuid: "eb30e33ed064429a8880af02a4ed7755"
format_stamp: "Formatted at 2026-08-05 22:52:22 on dist-test-slave-7lbf"
server_key: "151676bd8455030e61c0caa77c133b0f"
server_key_iv: "fc193cc29330929b954fb28dbc9104b5"
server_key_version: "encryptionkey@0"
I20260805 22:52:22.297251 745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:22.328173 745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:22.329160 745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:22.329452 745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:22.330595 745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:22.332299 745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:22.332406 745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:22.332530 745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:22.332605 745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:22.364168 745 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.2:37121
I20260805 22:52:22.364228 926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.2:37121 every 8 connection(s)
I20260805 22:52:22.365664 745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/data/info.pb
I20260805 22:52:22.371182 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 745
I20260805 22:52:22.371359 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/wal/instance
I20260805 22:52:22.371882 4208 external_mini_cluster.cc:1428] Setting key 3f3c5c97ae7f29244beae08d56391125
I20260805 22:52:22.374994 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.28.3:0
--local_ip_for_outbound_sockets=127.4.28.3
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37917
--builtin_ntp_servers=127.4.28.20:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 22:52:22.380940 742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.368671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.2:35903 (local address 127.4.28.62:37917)
0805 22:52:22.368855 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:22.368859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:22.370757 (+ 1898us) server_negotiation.cc:407] Connection header received
0805 22:52:22.373124 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:22.373131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:22.373233 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:22.373351 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:22.374763 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.375631 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.377580 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.378067 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.378215 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.379940 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:22.379946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:22.379950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:22.379976 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:22.380016 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:22.380031 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:22.380124 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:22.380409 (+ 285us) server_negotiation.cc:299] Negotiation successful
0805 22:52:22.380547 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 22:52:22.382261 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.369246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37917 (local address 127.4.28.2:35903)
0805 22:52:22.370503 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:22.370573 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:22.372731 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:22.373705 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:22.373716 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:22.373843 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:22.374548 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.374571 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.375849 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.375853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.377305 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.377354 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.378368 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.378379 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.378677 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.379507 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:52:22.379510 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:22.379562 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:52:22.379664 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:22.380140 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:22.380157 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:22.380167 (+ 10us) client_negotiation.cc:769] Sending connection context
0805 22:52:22.380376 (+ 209us) client_negotiation.cc:240] Negotiation successful
0805 22:52:22.380888 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":480,"threads_started":1}
I20260805 22:52:22.384279 927 heartbeater.cc:347] Connected to a master server at 127.4.28.62:37917
I20260805 22:52:22.384658 927 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:22.385671 927 heartbeater.cc:511] Master 127.4.28.62:37917 requested a full tablet report, sending...
I20260805 22:52:22.387594 462 ts_manager.cc:194] Registered new tserver with Master: eb30e33ed064429a8880af02a4ed7755 (127.4.28.2:37121)
I20260805 22:52:22.388669 462 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.2:35903
W20260805 22:52:22.749635 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:22.749980 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:22.750080 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:22.757663 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:22.757794 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:22.757911 931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.28.3
I20260805 22:52:22.767884 931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.28.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.28.3
--webserver_port=0
--tserver_master_addrs=127.4.28.62:37917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.28.3
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:22.769305 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:22.771334 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:22.785733 942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:22.785707 944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:22.786331 938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:22.787325 931 server_base.cc:1034] running on GCE node
I20260805 22:52:22.788228 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:22.789696 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:22.790918 931 hybrid_clock.cc:648] HybridClock initialized: now 1785970342790814 us; error 111 us; skew 500 ppm
I20260805 22:52:22.791313 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:52:22.794615 931 webserver.cc:460] Webserver started at http://127.4.28.3:33539/ using document root <none> and password file <none>
I20260805 22:52:22.795469 931 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:22.795619 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:22.795990 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:22.799279 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/data/instance:
uuid: "142b1ee89e764c1a9819bf17bd82af94"
format_stamp: "Formatted at 2026-08-05 22:52:22 on dist-test-slave-7lbf"
server_key: "1be72aee94a8ae98bcbf1a312a3231fd"
server_key_iv: "0de8eb860ac57f528024f227d26a92bf"
server_key_version: "encryptionkey@0"
I20260805 22:52:22.800329 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance:
uuid: "142b1ee89e764c1a9819bf17bd82af94"
format_stamp: "Formatted at 2026-08-05 22:52:22 on dist-test-slave-7lbf"
server_key: "1be72aee94a8ae98bcbf1a312a3231fd"
server_key_iv: "0de8eb860ac57f528024f227d26a92bf"
server_key_version: "encryptionkey@0"
I20260805 22:52:22.806310 931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260805 22:52:22.811452 957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:22.813217 931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.002s
I20260805 22:52:22.813424 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/wal
uuid: "142b1ee89e764c1a9819bf17bd82af94"
format_stamp: "Formatted at 2026-08-05 22:52:22 on dist-test-slave-7lbf"
server_key: "1be72aee94a8ae98bcbf1a312a3231fd"
server_key_iv: "0de8eb860ac57f528024f227d26a92bf"
server_key_version: "encryptionkey@0"
I20260805 22:52:22.813735 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:22.841259 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:22.842289 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:22.842628 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:22.843801 931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:52:22.845656 931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:52:22.845754 931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:22.845867 931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:52:22.845938 931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:22.881709 931 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.3:40225
I20260805 22:52:22.881793 1117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.3:40225 every 8 connection(s)
I20260805 22:52:22.883209 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/data/info.pb
I20260805 22:52:22.884119 740 heartbeater.cc:503] Master 127.4.28.62:37917 was elected leader, sending a full tablet report...
I20260805 22:52:22.888662 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 931
I20260805 22:52:22.888854 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/wal/instance
I20260805 22:52:22.889406 4208 external_mini_cluster.cc:1428] Setting key 31cd00c4be8284b29695301b00181bd7
I20260805 22:52:22.898576 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.886364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.3:41455 (local address 127.4.28.62:37917)
0805 22:52:22.886839 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:22.886844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:22.888193 (+ 1349us) server_negotiation.cc:407] Connection header received
0805 22:52:22.890722 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:22.890727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:22.890821 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:22.890939 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:22.892465 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.893436 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.895297 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.895739 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.895848 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.897419 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:22.897425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:22.897429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:22.897453 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:22.897500 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:22.897561 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:22.897653 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:22.898135 (+ 482us) server_negotiation.cc:299] Negotiation successful
0805 22:52:22.898240 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":228,"threads_started":1}
I20260805 22:52:22.899753 1122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.887011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:37917 (local address 127.4.28.3:41455)
0805 22:52:22.888012 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:22.888052 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:22.890353 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:22.891226 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:22.891237 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:22.891350 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:52:22.892261 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.892293 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.893635 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.893640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.895117 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.895135 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.895995 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.896001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.896237 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.897005 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 22:52:22.897009 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:52:22.897071 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 22:52:22.897179 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:52:22.897706 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:52:22.897718 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:52:22.897728 (+ 10us) client_negotiation.cc:769] Sending connection context
0805 22:52:22.898086 (+ 358us) client_negotiation.cc:240] Negotiation successful
0805 22:52:22.898489 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":299,"threads_started":1}
I20260805 22:52:22.901501 1118 heartbeater.cc:347] Connected to a master server at 127.4.28.62:37917
I20260805 22:52:22.901968 1118 heartbeater.cc:464] Registering TS with master...
I20260805 22:52:22.902839 1118 heartbeater.cc:511] Master 127.4.28.62:37917 requested a full tablet report, sending...
I20260805 22:52:22.904593 460 ts_manager.cc:194] Registered new tserver with Master: 142b1ee89e764c1a9819bf17bd82af94 (127.4.28.3:40225)
I20260805 22:52:22.905661 4208 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 22:52:22.905777 460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.28.3:41455
I20260805 22:52:22.916483 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.910454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.4.28.62:37917)
0805 22:52:22.910644 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:22.910649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:22.911137 (+ 488us) server_negotiation.cc:407] Connection header received
0805 22:52:22.911288 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:22.911292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:22.911358 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:22.911452 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:22.911912 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.912659 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.914213 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.914675 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.914788 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.915878 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:22.915883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:22.915887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:22.915909 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:22.915939 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:22.915951 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:22.916023 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:22.916136 (+ 113us) server_negotiation.cc:299] Negotiation successful
0805 22:52:22.916232 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 22:52:22.920928 460 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:52:22.928064 460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:52:22.967191 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.942111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:40225 (local address 127.0.0.1:48484)
0805 22:52:22.955167 (+ 13056us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:22.955266 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:22.955581 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:22.959080 (+ 3499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:22.959085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:22.959113 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:22.960011 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.960024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.962010 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.962016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.964480 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.964496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.966396 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.966402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.966584 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.966623 (+ 39us) client_negotiation.cc:769] Sending connection context
0805 22:52:22.966712 (+ 89us) client_negotiation.cc:240] Negotiation successful
0805 22:52:22.966851 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12629,"mutex_wait_us":515,"thread_start_us":13945,"threads_started":1}
I20260805 22:52:22.967127 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.942313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:37121 (local address 127.0.0.1:50574)
0805 22:52:22.955854 (+ 13541us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:22.955877 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:22.955992 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:22.959395 (+ 3403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:22.959404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:22.959429 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:22.959910 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.959925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.962343 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.962347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.964040 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.964057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.966003 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.966009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.966228 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.966294 (+ 66us) client_negotiation.cc:769] Sending connection context
0805 22:52:22.966468 (+ 174us) client_negotiation.cc:240] Negotiation successful
0805 22:52:22.966699 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13393,"mutex_wait_us":38,"thread_start_us":13565,"threads_started":1}
I20260805 22:52:22.967789 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.943885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.4.28.1:41635)
0805 22:52:22.957102 (+ 13217us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:22.957112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:22.957875 (+ 763us) server_negotiation.cc:407] Connection header received
0805 22:52:22.958005 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:22.958015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:22.958332 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:22.958572 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:22.959773 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.961218 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.964287 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.965872 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.966024 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.966320 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:22.967251 (+ 931us) server_negotiation.cc:299] Negotiation successful
0805 22:52:22.967445 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13014,"thread_start_us":12860,"threads_started":1}
I20260805 22:52:22.967900 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.942894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.4.28.2:37121)
0805 22:52:22.958133 (+ 15239us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:22.958142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:22.958177 (+ 35us) server_negotiation.cc:407] Connection header received
0805 22:52:22.958287 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:22.958294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:22.958547 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:22.959227 (+ 680us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:22.960136 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.962193 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.964326 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.965792 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.965935 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.966225 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:22.967372 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0805 22:52:22.967573 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15021,"thread_start_us":15445,"threads_started":1}
I20260805 22:52:22.967682 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.942439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.4.28.3:40225)
0805 22:52:22.958459 (+ 16020us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:22.958468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:22.958503 (+ 35us) server_negotiation.cc:407] Connection header received
0805 22:52:22.958608 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:22.958615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:22.958853 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:22.959062 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:22.960239 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.961860 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.964741 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.966195 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.966340 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.966630 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:22.966773 (+ 143us) server_negotiation.cc:299] Negotiation successful
0805 22:52:22.966956 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15828,"thread_start_us":15656,"threads_started":1}
I20260805 22:52:22.968297 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:22.943206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:41635 (local address 127.0.0.1:50954)
0805 22:52:22.957572 (+ 14366us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:22.957598 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:22.957723 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:22.958644 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:22.958653 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:22.958690 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:22.959559 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.959576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.961396 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.961402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.964002 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:22.964028 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:22.966471 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:22.966482 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:22.966698 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:22.966733 (+ 35us) client_negotiation.cc:769] Sending connection context
0805 22:52:22.966818 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 22:52:22.966955 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14211,"mutex_wait_us":116,"thread_start_us":14062,"threads_started":1}
I20260805 22:52:22.972239 634 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6cf942791e41e697f340fe002152c1 (DEFAULT_TABLE table=test-table [id=f09273c55a2b4de5bd2247c916a86085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:22.972561 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6cf942791e41e697f340fe002152c1 (DEFAULT_TABLE table=test-table [id=f09273c55a2b4de5bd2247c916a86085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:22.972548 832 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6cf942791e41e697f340fe002152c1 (DEFAULT_TABLE table=test-table [id=f09273c55a2b4de5bd2247c916a86085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 22:52:22.973841 634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6cf942791e41e697f340fe002152c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:22.974167 1025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6cf942791e41e697f340fe002152c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:22.974469 832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6cf942791e41e697f340fe002152c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:22.987489 1144 tablet_bootstrap.cc:492] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d: Bootstrap starting.
I20260805 22:52:22.989642 1145 tablet_bootstrap.cc:492] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: Bootstrap starting.
I20260805 22:52:22.993410 1145 tablet_bootstrap.cc:654] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:22.993427 1144 tablet_bootstrap.cc:654] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:22.994489 1146 tablet_bootstrap.cc:492] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: Bootstrap starting.
I20260805 22:52:22.995003 1144 log.cc:824] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:22.995003 1145 log.cc:824] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:22.998428 1146 tablet_bootstrap.cc:654] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:22.998906 1144 tablet_bootstrap.cc:492] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d: No bootstrap required, opened a new log
I20260805 22:52:22.999229 1144 ts_tablet_manager.cc:1397] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260805 22:52:22.999575 1146 log.cc:824] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:23.000165 1145 tablet_bootstrap.cc:492] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: No bootstrap required, opened a new log
I20260805 22:52:23.000463 1145 ts_tablet_manager.cc:1397] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260805 22:52:23.002543 1146 tablet_bootstrap.cc:492] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: No bootstrap required, opened a new log
I20260805 22:52:23.002790 1146 ts_tablet_manager.cc:1397] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260805 22:52:23.010468 1146 raft_consensus.cc:348] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } }
I20260805 22:52:23.010327 1144 raft_consensus.cc:348] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } }
I20260805 22:52:23.010862 1146 raft_consensus.cc:374] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:23.010859 1144 raft_consensus.cc:374] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:23.010962 1144 raft_consensus.cc:729] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77276d2c6bbf436488574b5b8e631b8d, State: Initialized, Role: FOLLOWER
I20260805 22:52:23.010957 1146 raft_consensus.cc:729] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb30e33ed064429a8880af02a4ed7755, State: Initialized, Role: FOLLOWER
I20260805 22:52:23.011601 1146 consensus_queue.cc:260] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } }
I20260805 22:52:23.011696 1144 consensus_queue.cc:260] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } }
I20260805 22:52:23.011639 1145 raft_consensus.cc:348] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } }
I20260805 22:52:23.012166 1145 raft_consensus.cc:374] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:23.012281 1145 raft_consensus.cc:729] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142b1ee89e764c1a9819bf17bd82af94, State: Initialized, Role: FOLLOWER
I20260805 22:52:23.012852 927 heartbeater.cc:503] Master 127.4.28.62:37917 was elected leader, sending a full tablet report...
I20260805 22:52:23.013043 1145 consensus_queue.cc:260] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } }
I20260805 22:52:23.014273 1118 heartbeater.cc:503] Master 127.4.28.62:37917 was elected leader, sending a full tablet report...
I20260805 22:52:23.015069 1145 ts_tablet_manager.cc:1428] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260805 22:52:23.015120 1146 ts_tablet_manager.cc:1428] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260805 22:52:23.017002 1144 ts_tablet_manager.cc:1428] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d: Time spent starting tablet: real 0.018s user 0.004s sys 0.012s
W20260805 22:52:23.107156 741 tablet.cc:2369] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:52:23.118155 929 tablet.cc:2369] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 22:52:23.135428 1119 tablet.cc:2369] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:52:23.304442 1152 raft_consensus.cc:482] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:52:23.304785 1152 raft_consensus.cc:504] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } }
I20260805 22:52:23.306655 1152 leader_election.cc:288] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 142b1ee89e764c1a9819bf17bd82af94 (127.4.28.3:40225), eb30e33ed064429a8880af02a4ed7755 (127.4.28.2:37121)
I20260805 22:52:23.314338 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.307212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.2:37121 (local address 127.4.28.1:53175)
0805 22:52:23.308574 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:23.308590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:23.308729 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:23.309069 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:23.309073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:23.309101 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:23.309420 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:23.309429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.310478 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.310483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:23.312251 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:23.312266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.313453 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.313460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:23.313722 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:23.313784 (+ 62us) client_negotiation.cc:769] Sending connection context
0805 22:52:23.313942 (+ 158us) client_negotiation.cc:240] Negotiation successful
0805 22:52:23.314057 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":166,"thread_start_us":930,"threads_started":1}
I20260805 22:52:23.314324 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.308338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:36031 (local address 127.4.28.3:40225)
0805 22:52:23.308575 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:23.308580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:23.308596 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:52:23.308659 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:23.308663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:23.308720 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:23.308797 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:23.309400 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.310411 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.312491 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.313260 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.313378 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:23.313479 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:23.313942 (+ 463us) server_negotiation.cc:299] Negotiation successful
0805 22:52:23.314079 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260805 22:52:23.314361 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.307249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.28.1:53175 (local address 127.4.28.2:37121)
0805 22:52:23.307474 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:23.307482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:23.308696 (+ 1214us) server_negotiation.cc:407] Connection header received
0805 22:52:23.308869 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:23.308873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:23.308949 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:23.309025 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 22:52:23.309604 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.310330 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.312467 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.313238 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.313368 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:23.313469 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:23.313985 (+ 516us) server_negotiation.cc:299] Negotiation successful
0805 22:52:23.314119 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260805 22:52:23.315011 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.307179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.3:40225 (local address 127.4.28.1:36031)
0805 22:52:23.308161 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:23.308191 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0805 22:52:23.308387 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:52:23.308843 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:52:23.308847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:52:23.308872 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 22:52:23.309239 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:23.309249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.310557 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.310562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:23.312252 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:52:23.312267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.313471 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.313477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:52:23.313717 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:23.313776 (+ 59us) client_negotiation.cc:769] Sending connection context
0805 22:52:23.314553 (+ 777us) client_negotiation.cc:240] Negotiation successful
0805 22:52:23.314690 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":60,"thread_start_us":493,"threads_started":1}
I20260805 22:52:23.315632 865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6cf942791e41e697f340fe002152c1" candidate_uuid: "77276d2c6bbf436488574b5b8e631b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb30e33ed064429a8880af02a4ed7755" is_pre_election: true
I20260805 22:52:23.316217 865 raft_consensus.cc:2454] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77276d2c6bbf436488574b5b8e631b8d in term 0.
I20260805 22:52:23.316188 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6cf942791e41e697f340fe002152c1" candidate_uuid: "77276d2c6bbf436488574b5b8e631b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142b1ee89e764c1a9819bf17bd82af94" is_pre_election: true
I20260805 22:52:23.316720 1050 raft_consensus.cc:2454] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77276d2c6bbf436488574b5b8e631b8d in term 0.
I20260805 22:52:23.317004 570 leader_election.cc:302] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77276d2c6bbf436488574b5b8e631b8d, eb30e33ed064429a8880af02a4ed7755; no voters:
I20260805 22:52:23.317533 1152 raft_consensus.cc:2781] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:52:23.317708 1152 raft_consensus.cc:482] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:52:23.317828 1152 raft_consensus.cc:3037] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:23.320147 1152 raft_consensus.cc:504] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } }
I20260805 22:52:23.320809 1152 leader_election.cc:288] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [CANDIDATE]: Term 1 election: Requested vote from peers 142b1ee89e764c1a9819bf17bd82af94 (127.4.28.3:40225), eb30e33ed064429a8880af02a4ed7755 (127.4.28.2:37121)
I20260805 22:52:23.321462 865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6cf942791e41e697f340fe002152c1" candidate_uuid: "77276d2c6bbf436488574b5b8e631b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb30e33ed064429a8880af02a4ed7755"
I20260805 22:52:23.321493 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6cf942791e41e697f340fe002152c1" candidate_uuid: "77276d2c6bbf436488574b5b8e631b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142b1ee89e764c1a9819bf17bd82af94"
I20260805 22:52:23.321730 865 raft_consensus.cc:3037] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:23.321798 1050 raft_consensus.cc:3037] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:23.324205 865 raft_consensus.cc:2454] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77276d2c6bbf436488574b5b8e631b8d in term 1.
I20260805 22:52:23.324254 1050 raft_consensus.cc:2454] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77276d2c6bbf436488574b5b8e631b8d in term 1.
I20260805 22:52:23.324829 571 leader_election.cc:302] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 142b1ee89e764c1a9819bf17bd82af94, 77276d2c6bbf436488574b5b8e631b8d, eb30e33ed064429a8880af02a4ed7755; no voters:
I20260805 22:52:23.325263 1152 raft_consensus.cc:2781] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:23.325831 1152 raft_consensus.cc:686] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 1 LEADER]: Becoming Leader. State: Replica: 77276d2c6bbf436488574b5b8e631b8d, State: Running, Role: LEADER
I20260805 22:52:23.326435 1152 consensus_queue.cc:237] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } }
I20260805 22:52:23.331365 460 catalog_manager.cc:5495] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 77276d2c6bbf436488574b5b8e631b8d (127.4.28.1). New cstate: current_term: 1 leader_uuid: "77276d2c6bbf436488574b5b8e631b8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77276d2c6bbf436488574b5b8e631b8d" member_type: VOTER last_known_addr { host: "127.4.28.1" port: 41635 } health_report { overall_health: HEALTHY } } }
I20260805 22:52:23.428427 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.422698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.4.28.1:41635)
0805 22:52:23.422891 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:23.422896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:23.423225 (+ 329us) server_negotiation.cc:407] Connection header received
0805 22:52:23.423427 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:23.423430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:23.423538 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:23.423679 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:52:23.424041 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.424750 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.426340 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.426786 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.426888 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:23.427392 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:52:23.427933 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:52:23.428051 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:23.428136 (+ 85us) server_negotiation.cc:299] Negotiation successful
0805 22:52:23.428214 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 22:52:23.437945 865 raft_consensus.cc:1261] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [term 1 FOLLOWER]: Refusing update from remote peer 77276d2c6bbf436488574b5b8e631b8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:23.437935 1050 raft_consensus.cc:1261] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [term 1 FOLLOWER]: Refusing update from remote peer 77276d2c6bbf436488574b5b8e631b8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:52:23.438989 1152 consensus_queue.cc:1035] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb30e33ed064429a8880af02a4ed7755" member_type: VOTER last_known_addr { host: "127.4.28.2" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:23.439358 1158 consensus_queue.cc:1035] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "142b1ee89e764c1a9819bf17bd82af94" member_type: VOTER last_known_addr { host: "127.4.28.3" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:52:23.450376 1163 mvcc.cc:205] Tried to move back new op lower bound from 7315334526709760000 to 7315334526269464576. Current Snapshot: MvccSnapshot[applied={T|T < 7315334526709760000}]
I20260805 22:52:23.450977 1165 mvcc.cc:205] Tried to move back new op lower bound from 7315334526709760000 to 7315334526269464576. Current Snapshot: MvccSnapshot[applied={T|T < 7315334526709760000}]
I20260805 22:52:23.469583 460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:23.469887 460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 22:52:23.473927 460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c753ee53bce4852a77e01b6fef9e23e: Sending DeleteTablet for 3 replicas of tablet 3f6cf942791e41e697f340fe002152c1
I20260805 22:52:23.475317 634 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6cf942791e41e697f340fe002152c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:23 UTC) from {username='slave'} at 127.0.0.1:50954
I20260805 22:52:23.475354 832 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6cf942791e41e697f340fe002152c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:23 UTC) from {username='slave'} at 127.0.0.1:50574
I20260805 22:52:23.475698 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6cf942791e41e697f340fe002152c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 22:52:23 UTC) from {username='slave'} at 127.0.0.1:48484
I20260805 22:52:23.477491 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 548
I20260805 22:52:23.489465 1176 tablet_replica.cc:330] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: stopping tablet replica
I20260805 22:52:23.490279 1176 raft_consensus.cc:2229] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:23.491060 1176 raft_consensus.cc:2258] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:23.494107 1176 ts_tablet_manager.cc:1905] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:23.503346 1176 ts_tablet_manager.cc:1918] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:23.503746 1176 log.cc:1196] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-1/wal/wals/3f6cf942791e41e697f340fe002152c1
I20260805 22:52:23.504505 1176 ts_tablet_manager.cc:1939] T 3f6cf942791e41e697f340fe002152c1 P eb30e33ed064429a8880af02a4ed7755: Deleting consensus metadata
I20260805 22:52:23.507004 437 catalog_manager.cc:4841] TS eb30e33ed064429a8880af02a4ed7755 (127.4.28.2:37121): tablet 3f6cf942791e41e697f340fe002152c1 (table test-table [id=f09273c55a2b4de5bd2247c916a86085]) successfully deleted
I20260805 22:52:23.510278 1180 tablet_replica.cc:330] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d: stopping tablet replica
I20260805 22:52:23.511085 1180 raft_consensus.cc:2229] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:52:23.511979 1180 raft_consensus.cc:2258] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:23.513003 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.479659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.4.28.1:41635)
0805 22:52:23.479902 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:23.479910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:23.501353 (+ 21443us) server_negotiation.cc:407] Connection header received
0805 22:52:23.502153 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:23.502162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:23.502325 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:23.502410 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:23.503363 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.504715 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.507644 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.508432 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.508572 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:23.510617 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:23.510633 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:23.510658 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:23.511674 (+ 1016us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:23.511766 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:23.511801 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:23.511936 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:23.512272 (+ 336us) server_negotiation.cc:299] Negotiation successful
0805 22:52:23.512421 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260805 22:52:23.513381 1181 tablet_replica.cc:330] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: stopping tablet replica
I20260805 22:52:23.513948 1181 raft_consensus.cc:2229] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:52:23.514381 730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:23.514590 1181 raft_consensus.cc:2258] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:52:23.517481 1181 ts_tablet_manager.cc:1905] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:23.525629 1181 ts_tablet_manager.cc:1918] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 22:52:23.525874 1181 log.cc:1196] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: Deleting WAL directory at /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785970002413767-4208-0/minicluster-data/ts-2/wal/wals/3f6cf942791e41e697f340fe002152c1
I20260805 22:52:23.526445 1181 ts_tablet_manager.cc:1939] T 3f6cf942791e41e697f340fe002152c1 P 142b1ee89e764c1a9819bf17bd82af94: Deleting consensus metadata
I20260805 22:52:23.528825 438 catalog_manager.cc:4841] TS 142b1ee89e764c1a9819bf17bd82af94 (127.4.28.3:40225): tablet 3f6cf942791e41e697f340fe002152c1 (table test-table [id=f09273c55a2b4de5bd2247c916a86085]) successfully deleted
I20260805 22:52:23.515708 1180 ts_tablet_manager.cc:1905] T 3f6cf942791e41e697f340fe002152c1 P 77276d2c6bbf436488574b5b8e631b8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 22:52:23.620652 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 548
W20260805 22:52:23.645759 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.28.1:41635 (error 108)
I20260805 22:52:23.645983 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 745
I20260805 22:52:23.648115 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.647305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:41635 (local address 127.0.0.1:50978)
0805 22:52:23.647793 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:23.647909 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":244,"threads_started":1}
W20260805 22:52:23.648393 439 catalog_manager.cc:4584] TS 77276d2c6bbf436488574b5b8e631b8d (127.4.28.1:41635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f6cf942791e41e697f340fe002152c1: Network error: Client connection negotiation failed: client connection to 127.4.28.1:41635: connect: Connection refused (error 111)
I20260805 22:52:23.653946 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.646984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.4.28.2:37121)
0805 22:52:23.647178 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:23.647183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:23.647204 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:52:23.647407 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:23.647410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:23.647502 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:23.647604 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:23.648099 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.649075 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.650526 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.650978 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.651104 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:23.652327 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:23.652338 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:23.652356 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:23.653038 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:23.653099 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:23.653125 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:23.653247 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:23.653416 (+ 169us) server_negotiation.cc:299] Negotiation successful
0805 22:52:23.653587 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:52:23.654723 922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:23.702672 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.702140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:41635 (local address 127.0.0.1:50988)
0805 22:52:23.702396 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:23.702505 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260805 22:52:23.703925 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.703490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:41635 (local address 127.0.0.1:51004)
0805 22:52:23.703675 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:23.703771 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260805 22:52:23.748375 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 745
I20260805 22:52:23.764649 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.764092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:41635 (local address 127.0.0.1:51008)
0805 22:52:23.764357 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:23.764476 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260805 22:52:23.765928 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.765490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:41635 (local address 127.0.0.1:51018)
0805 22:52:23.765682 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:23.765766 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260805 22:52:23.774370 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 931
I20260805 22:52:23.781877 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.775303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.4.28.3:40225)
0805 22:52:23.775509 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:23.775514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:23.775536 (+ 22us) server_negotiation.cc:407] Connection header received
0805 22:52:23.775735 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:23.775741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:23.775874 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:23.775977 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:23.776517 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.777328 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.778714 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:23.779186 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:23.779289 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:23.780380 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:23.780390 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:23.780422 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:23.781054 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:23.781108 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:23.781131 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:23.781232 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:23.781337 (+ 105us) server_negotiation.cc:299] Negotiation successful
0805 22:52:23.781451 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 22:52:23.782656 1112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:23.846879 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.846309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:41635 (local address 127.0.0.1:51032)
0805 22:52:23.846598 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:23.846721 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260805 22:52:23.848181 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:23.847641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.1:41635 (local address 127.0.0.1:51034)
0805 22:52:23.847914 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0805 22:52:23.848014 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.1:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260805 22:52:23.873983 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 931
I20260805 22:52:23.899106 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 412
I20260805 22:52:23.900314 503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:24.013384 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 412
2026-08-05T22:52:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3367 ms)
[----------] 6 tests from EncryptionPolicyTest (24637 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:24 dist-test-slave-7lbf krb5kdc[1200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:24 dist-test-slave-7lbf krb5kdc[1200](info): set up 2 sockets
Aug 05 22:52:24 dist-test-slave-7lbf krb5kdc[1200](info): commencing operation
krb5kdc: starting...
W20260805 22:52:25.261157 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.001s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:25 dist-test-slave-7lbf krb5kdc[1200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:25.627943 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:26.001693 1215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:26.002060 1215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:26.002208 1215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:26.009826 1215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:26.009987 1215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:26.010070 1215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:26.010164 1215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:26.010224 1215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:26.021034 1215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:26.022598 1215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:26.024102 1215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:26.037109 1221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:26.037282 1225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:26.037330 1222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:26.038384 1215 server_base.cc:1034] running on GCE node
I20260805 22:52:26.039290 1215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:26.040921 1215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:26.042140 1215 hybrid_clock.cc:648] HybridClock initialized: now 1785970346042088 us; error 49 us; skew 500 ppm
I20260805 22:52:26.042557 1215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:26 dist-test-slave-7lbf krb5kdc[1200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:26.053164 1215 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:26.055194 1215 webserver.cc:460] Webserver started at http://127.0.0.1:40969/ using document root <none> and password file <none>
I20260805 22:52:26.056031 1215 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:26.056183 1215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:26.056535 1215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:26.060125 1215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "6eb64c03ba6c4850b832619103a8e324"
format_stamp: "Formatted at 2026-08-05 22:52:26 on dist-test-slave-7lbf"
server_key: "d5f6ca5c1a112f91c15394b6d66776b9"
server_key_iv: "a8a8c8b9d55edc76ee855b89d47ad641"
server_key_version: "encryptionkey@0"
I20260805 22:52:26.061050 1215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "6eb64c03ba6c4850b832619103a8e324"
format_stamp: "Formatted at 2026-08-05 22:52:26 on dist-test-slave-7lbf"
server_key: "d5f6ca5c1a112f91c15394b6d66776b9"
server_key_iv: "a8a8c8b9d55edc76ee855b89d47ad641"
server_key_version: "encryptionkey@0"
I20260805 22:52:26.067106 1215 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260805 22:52:26.071521 1234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:26.073112 1215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 22:52:26.073321 1215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "6eb64c03ba6c4850b832619103a8e324"
format_stamp: "Formatted at 2026-08-05 22:52:26 on dist-test-slave-7lbf"
server_key: "d5f6ca5c1a112f91c15394b6d66776b9"
server_key_iv: "a8a8c8b9d55edc76ee855b89d47ad641"
server_key_version: "encryptionkey@0"
I20260805 22:52:26.073628 1215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:26.088886 1215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:26.093847 1215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:26.094172 1215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:26.114322 1215 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45763
I20260805 22:52:26.114355 1306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45763 every 8 connection(s)
I20260805 22:52:26.115880 1215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:26.120018 1307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:26.125828 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 1215
I20260805 22:52:26.126076 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:26.126883 4208 external_mini_cluster.cc:1428] Setting key ffdce076303b05bbeb79be9cfc4d5c93
I20260805 22:52:26.132782 1307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324: Bootstrap starting.
I20260805 22:52:26.136965 1307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:26.138522 1307 log.cc:824] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:26.142745 1307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324: No bootstrap required, opened a new log
Aug 05 22:52:26 dist-test-slave-7lbf krb5kdc[1200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260805 22:52:26.152249 1307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb64c03ba6c4850b832619103a8e324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } }
I20260805 22:52:26.152864 1307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:26.153000 1307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb64c03ba6c4850b832619103a8e324, State: Initialized, Role: FOLLOWER
I20260805 22:52:26.153824 1307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb64c03ba6c4850b832619103a8e324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } }
I20260805 22:52:26.154100 1307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:26.154247 1307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:26.154426 1307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:26.154842 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:26.129223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51595 (local address 127.0.0.1:45763)
0805 22:52:26.130261 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:26.130276 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:26.130329 (+ 53us) server_negotiation.cc:407] Connection header received
0805 22:52:26.132207 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:26.132235 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:26.132743 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:26.133167 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:26.135228 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:26.136661 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:26.138806 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:26.139967 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:26.140201 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:26.145596 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:26.145609 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:26.145630 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:26.145688 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:26.149248 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:26.150637 (+ 1389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:26.150650 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:26.150661 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:26.150886 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:26.151461 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:26.151465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:26.151468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:26.152138 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:26.152428 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:26.153097 (+ 669us) server_negotiation.cc:299] Negotiation successful
0805 22:52:26.153587 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":475,"threads_started":1}
I20260805 22:52:26.157034 1307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb64c03ba6c4850b832619103a8e324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } }
I20260805 22:52:26.157626 1307 leader_election.cc:302] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eb64c03ba6c4850b832619103a8e324; no voters:
I20260805 22:52:26.158345 1307 leader_election.cc:288] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:26.158496 1312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:26.159235 1312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [term 1 LEADER]: Becoming Leader. State: Replica: 6eb64c03ba6c4850b832619103a8e324, State: Running, Role: LEADER
I20260805 22:52:26.160015 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:26.159891 1312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb64c03ba6c4850b832619103a8e324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } }
I20260805 22:52:26.161506 1307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:26.162979 1314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eb64c03ba6c4850b832619103a8e324. Latest consensus state: current_term: 1 leader_uuid: "6eb64c03ba6c4850b832619103a8e324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb64c03ba6c4850b832619103a8e324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } } }
I20260805 22:52:26.163424 1314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:26.164064 1315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eb64c03ba6c4850b832619103a8e324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb64c03ba6c4850b832619103a8e324" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } } }
I20260805 22:52:26.164389 1315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:26.166055 1323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:26.172806 1323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:26.196730 1323 catalog_manager.cc:1347] Generated new cluster ID: 47a020671c9546b99abf240f7ce237b2
I20260805 22:52:26.196875 1323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:26.201040 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:26.170091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43357 (local address 127.0.0.1:45763)
0805 22:52:26.170308 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:26.170314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:26.181123 (+ 10809us) server_negotiation.cc:407] Connection header received
0805 22:52:26.181772 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:26.181779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:26.181873 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:26.181983 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:26.184399 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:26.185693 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:26.187730 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:26.188435 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:26.188580 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:26.192338 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:26.192347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:26.192352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:26.192498 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:26.196592 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:26.197947 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:26.197955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:26.197958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:26.198147 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:26.198855 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:26.198861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:26.198865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:26.199319 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:26.199611 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:26.200409 (+ 798us) server_negotiation.cc:299] Negotiation successful
0805 22:52:26.200645 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260805 22:52:26.216645 1323 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:26.218158 1323 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:26.233470 1323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eb64c03ba6c4850b832619103a8e324: Generated new TSK 0
I20260805 22:52:26.234815 1323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:26.246693 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 1215
I20260805 22:52:26.248420 1301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:26.344099 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 1215
W20260805 22:52:26.370718 1192 connection.cc:535] client connection to 127.0.0.1:45763 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45763): Connection reset by peer (error 104)
2026-08-05T22:52:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:26 dist-test-slave-7lbf krb5kdc[1358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:26 dist-test-slave-7lbf krb5kdc[1358](info): set up 2 sockets
Aug 05 22:52:26 dist-test-slave-7lbf krb5kdc[1358](info): commencing operation
krb5kdc: starting...
W20260805 22:52:27.589670 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:27 dist-test-slave-7lbf krb5kdc[1358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:27.958138 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:28.335696 1376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:28.336038 1376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:28.336184 1376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:28.343880 1376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:28.344009 1376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:28.344096 1376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:28.344182 1376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:28.344241 1376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:28.354894 1376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:28.356371 1376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:28.357863 1376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:28.371281 1382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:28.371322 1384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:28.371413 1390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:28.371863 1376 server_base.cc:1034] running on GCE node
I20260805 22:52:28.373097 1376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:28.374941 1376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:28.376189 1376 hybrid_clock.cc:648] HybridClock initialized: now 1785970348376122 us; error 63 us; skew 500 ppm
I20260805 22:52:28.376580 1376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:28 dist-test-slave-7lbf krb5kdc[1358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:28.380275 1376 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:28.382099 1376 webserver.cc:460] Webserver started at http://127.0.0.1:32785/ using document root <none> and password file <none>
I20260805 22:52:28.382887 1376 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:28.383040 1376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:28.383380 1376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:28.386919 1376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "319d4d9290b0485f86926549fa7e8c51"
format_stamp: "Formatted at 2026-08-05 22:52:28 on dist-test-slave-7lbf"
server_key: "9e4c5a6fab2a25f8b09ca7d826b2fe69"
server_key_iv: "37c7bad960d3eca4baa08f907c34c83f"
server_key_version: "encryptionkey@0"
I20260805 22:52:28.387784 1376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "319d4d9290b0485f86926549fa7e8c51"
format_stamp: "Formatted at 2026-08-05 22:52:28 on dist-test-slave-7lbf"
server_key: "9e4c5a6fab2a25f8b09ca7d826b2fe69"
server_key_iv: "37c7bad960d3eca4baa08f907c34c83f"
server_key_version: "encryptionkey@0"
I20260805 22:52:28.393935 1376 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260805 22:52:28.398277 1397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:28.399900 1376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:52:28.400151 1376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "319d4d9290b0485f86926549fa7e8c51"
format_stamp: "Formatted at 2026-08-05 22:52:28 on dist-test-slave-7lbf"
server_key: "9e4c5a6fab2a25f8b09ca7d826b2fe69"
server_key_iv: "37c7bad960d3eca4baa08f907c34c83f"
server_key_version: "encryptionkey@0"
I20260805 22:52:28.400377 1376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:28.420262 1376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:28.425334 1376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:28.425729 1376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:28.445171 1376 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41645
I20260805 22:52:28.445194 1477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41645 every 8 connection(s)
I20260805 22:52:28.446751 1376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:28.450876 1478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:28.456436 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 1376
I20260805 22:52:28.456614 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:28.457099 4208 external_mini_cluster.cc:1428] Setting key b466704581000fd29ab68df20c98d443
I20260805 22:52:28.462780 1478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51: Bootstrap starting.
I20260805 22:52:28.466992 1478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:28.468430 1478 log.cc:824] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:28.472074 1478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51: No bootstrap required, opened a new log
I20260805 22:52:28.473256 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:28.459253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46023 (local address 127.0.0.1:41645)
0805 22:52:28.460403 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:28.460417 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:28.460491 (+ 74us) server_negotiation.cc:407] Connection header received
0805 22:52:28.462560 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:28.462591 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:28.463128 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:28.463560 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:28.464331 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:28.465927 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:28.468015 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:28.468826 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:28.469025 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:28.470506 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:28.470516 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:28.470553 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:28.471187 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:28.471286 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:28.471326 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:28.471493 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:28.471882 (+ 389us) server_negotiation.cc:299] Negotiation successful
0805 22:52:28.472093 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":568,"threads_started":1}
I20260805 22:52:28.477617 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:28.481415 1478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319d4d9290b0485f86926549fa7e8c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } }
I20260805 22:52:28.482030 1478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:28.482167 1478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319d4d9290b0485f86926549fa7e8c51, State: Initialized, Role: FOLLOWER
I20260805 22:52:28.482910 1478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319d4d9290b0485f86926549fa7e8c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } }
I20260805 22:52:28.483203 1478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:28.483362 1478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:28.483556 1478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:28.486701 1478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319d4d9290b0485f86926549fa7e8c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } }
I20260805 22:52:28.487282 1478 leader_election.cc:302] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 319d4d9290b0485f86926549fa7e8c51; no voters:
I20260805 22:52:28.488126 1478 leader_election.cc:288] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:28.488255 1496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:28.489543 1496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [term 1 LEADER]: Becoming Leader. State: Replica: 319d4d9290b0485f86926549fa7e8c51, State: Running, Role: LEADER
I20260805 22:52:28.490288 1496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319d4d9290b0485f86926549fa7e8c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } }
I20260805 22:52:28.490688 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:28.483011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43739 (local address 127.0.0.1:41645)
0805 22:52:28.483226 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:28.483233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:28.483739 (+ 506us) server_negotiation.cc:407] Connection header received
0805 22:52:28.483945 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:28.483951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:28.484046 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:28.484181 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:28.484722 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:28.485881 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:28.487379 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:28.488117 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:28.488263 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:28.489799 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:28.489806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:28.489812 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:28.489839 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:28.489893 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:28.489914 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:28.490033 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:28.490261 (+ 228us) server_negotiation.cc:299] Negotiation successful
0805 22:52:28.490323 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:52:28.491050 1478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:28.494130 1498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 319d4d9290b0485f86926549fa7e8c51. Latest consensus state: current_term: 1 leader_uuid: "319d4d9290b0485f86926549fa7e8c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319d4d9290b0485f86926549fa7e8c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } } }
I20260805 22:52:28.494143 1497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "319d4d9290b0485f86926549fa7e8c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319d4d9290b0485f86926549fa7e8c51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } } }
I20260805 22:52:28.494663 1497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:28.494663 1498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:28.502388 1504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:28.508670 1504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:28.518642 1504 catalog_manager.cc:1347] Generated new cluster ID: 3a7b216787a646f88651dc0b46832560
I20260805 22:52:28.518790 1504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:28.532955 1504 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:28.534472 1504 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:28.544929 1504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 319d4d9290b0485f86926549fa7e8c51: Generated new TSK 0
I20260805 22:52:28.545930 1504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:28.556547 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 1376
W20260805 22:52:28.558208 1469 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46023
W20260805 22:52:28.559072 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 1376: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:28.559242 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 1376
W20260805 22:52:28.584636 1356 connection.cc:535] client connection to 127.0.0.1:41645 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41645): Connection reset by peer (error 104)
2026-08-05T22:52:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:28 dist-test-slave-7lbf krb5kdc[1524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:28 dist-test-slave-7lbf krb5kdc[1524](info): set up 2 sockets
Aug 05 22:52:28 dist-test-slave-7lbf krb5kdc[1524](info): commencing operation
krb5kdc: starting...
W20260805 22:52:29.803035 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.005s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:30 dist-test-slave-7lbf krb5kdc[1524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:30.170060 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:30.541612 1543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:30.541991 1543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:30.542138 1543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:30.550043 1543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:30.550215 1543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:30.550304 1543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:30.550391 1543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:30.550451 1543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:30.562220 1543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44389
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:30.563678 1543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:30.565176 1543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:30.577674 1550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:30.577756 1551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:30.577916 1553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:30.578728 1543 server_base.cc:1034] running on GCE node
I20260805 22:52:30.579587 1543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:30.581161 1543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:30.582436 1543 hybrid_clock.cc:648] HybridClock initialized: now 1785970350582361 us; error 53 us; skew 500 ppm
I20260805 22:52:30.582954 1543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:30 dist-test-slave-7lbf krb5kdc[1524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:30.586658 1543 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:30.588562 1543 webserver.cc:460] Webserver started at http://127.0.0.1:45973/ using document root <none> and password file <none>
I20260805 22:52:30.589386 1543 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:30.589571 1543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:30.589963 1543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:30.593562 1543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "f3e535b08a97408ca1259c0e7e855a66"
format_stamp: "Formatted at 2026-08-05 22:52:30 on dist-test-slave-7lbf"
server_key: "a31601116d9bea932c7280e2b827a74d"
server_key_iv: "94f40f3619227a0486c8991f85bdfd3c"
server_key_version: "encryptionkey@0"
I20260805 22:52:30.594563 1543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "f3e535b08a97408ca1259c0e7e855a66"
format_stamp: "Formatted at 2026-08-05 22:52:30 on dist-test-slave-7lbf"
server_key: "a31601116d9bea932c7280e2b827a74d"
server_key_iv: "94f40f3619227a0486c8991f85bdfd3c"
server_key_version: "encryptionkey@0"
I20260805 22:52:30.601016 1543 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260805 22:52:30.605314 1562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:30.607079 1543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:52:30.607291 1543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "f3e535b08a97408ca1259c0e7e855a66"
format_stamp: "Formatted at 2026-08-05 22:52:30 on dist-test-slave-7lbf"
server_key: "a31601116d9bea932c7280e2b827a74d"
server_key_iv: "94f40f3619227a0486c8991f85bdfd3c"
server_key_version: "encryptionkey@0"
I20260805 22:52:30.607515 1543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:30.614215 1543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:30.619122 1543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:30.619453 1543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:30.639129 1543 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44389
I20260805 22:52:30.639179 1630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44389 every 8 connection(s)
I20260805 22:52:30.640719 1543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:30.644986 1632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:30.646931 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 1543
I20260805 22:52:30.647140 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:30.647713 4208 external_mini_cluster.cc:1428] Setting key 893c2b3b47b1c0b90658aac8920d8d67
I20260805 22:52:30.657264 1636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:30.650009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40465 (local address 127.0.0.1:44389)
0805 22:52:30.651044 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:30.651060 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:30.651119 (+ 59us) server_negotiation.cc:407] Connection header received
0805 22:52:30.653022 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:30.653061 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:30.653592 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:30.654206 (+ 614us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:30.655414 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:30.655422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:30.655438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:30.655457 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:30.655527 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:30.655555 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:30.655796 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:30.655941 (+ 145us) server_negotiation.cc:299] Negotiation successful
0805 22:52:30.656169 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":456,"threads_started":1}
I20260805 22:52:30.660454 1632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66: Bootstrap starting.
I20260805 22:52:30.662137 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:30.664448 1632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:30.666064 1632 log.cc:824] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:30.669793 1636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:30.666948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55251 (local address 127.0.0.1:44389)
0805 22:52:30.667150 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:30.667157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:30.667542 (+ 385us) server_negotiation.cc:407] Connection header received
0805 22:52:30.667721 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:30.667726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:30.667781 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:30.667871 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:30.669063 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:30.669067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:30.669071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:30.669091 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:30.669140 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:30.669158 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:30.669278 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:30.669419 (+ 141us) server_negotiation.cc:299] Negotiation successful
0805 22:52:30.669492 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:52:30.669942 1632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66: No bootstrap required, opened a new log
I20260805 22:52:30.678479 1632 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e535b08a97408ca1259c0e7e855a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } }
I20260805 22:52:30.678941 1632 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:30.679050 1632 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e535b08a97408ca1259c0e7e855a66, State: Initialized, Role: FOLLOWER
I20260805 22:52:30.679736 1632 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e535b08a97408ca1259c0e7e855a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } }
I20260805 22:52:30.679983 1632 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:30.680132 1632 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:30.680298 1632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:30.682978 1632 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e535b08a97408ca1259c0e7e855a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } }
I20260805 22:52:30.683645 1632 leader_election.cc:302] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3e535b08a97408ca1259c0e7e855a66; no voters:
I20260805 22:52:30.684405 1632 leader_election.cc:288] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:30.684561 1646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:30.685261 1646 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [term 1 LEADER]: Becoming Leader. State: Replica: f3e535b08a97408ca1259c0e7e855a66, State: Running, Role: LEADER
I20260805 22:52:30.685886 1646 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e535b08a97408ca1259c0e7e855a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } }
I20260805 22:52:30.686637 1632 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:30.689209 1648 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e535b08a97408ca1259c0e7e855a66. Latest consensus state: current_term: 1 leader_uuid: "f3e535b08a97408ca1259c0e7e855a66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e535b08a97408ca1259c0e7e855a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } } }
I20260805 22:52:30.689179 1647 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3e535b08a97408ca1259c0e7e855a66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e535b08a97408ca1259c0e7e855a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } } }
I20260805 22:52:30.689759 1647 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:30.689759 1648 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:30.698340 1658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:30.704891 1658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:30.714845 1658 catalog_manager.cc:1347] Generated new cluster ID: 6193644dcc7c4b5b9e18678bf144c08f
I20260805 22:52:30.714986 1658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:30.731072 1658 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:30.731570 1658 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:30.739610 1658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3e535b08a97408ca1259c0e7e855a66: Generated new TSK 0
I20260805 22:52:30.740578 1658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:30.747023 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 1543
W20260805 22:52:30.748282 1623 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40465
W20260805 22:52:30.749106 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 1543: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:30.749265 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 1543
2026-08-05T22:52:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:30 dist-test-slave-7lbf krb5kdc[1675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:30 dist-test-slave-7lbf krb5kdc[1675](info): set up 2 sockets
Aug 05 22:52:30 dist-test-slave-7lbf krb5kdc[1675](info): commencing operation
krb5kdc: starting...
W20260805 22:52:31.987304 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.007s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:32 dist-test-slave-7lbf krb5kdc[1675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:32.353039 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43303
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:32.723702 1693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:32.724053 1693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:32.724185 1693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:32.731963 1693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:32.732101 1693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:32.732187 1693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:32.732268 1693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:32.732333 1693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:32.742901 1693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:32.744697 1693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:32.746351 1693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:32.760084 1703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:32.760129 1700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:32.760155 1705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:32.761186 1693 server_base.cc:1034] running on GCE node
I20260805 22:52:32.762161 1693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:32.763873 1693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:32.765130 1693 hybrid_clock.cc:648] HybridClock initialized: now 1785970352765078 us; error 55 us; skew 500 ppm
I20260805 22:52:32.765570 1693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:32 dist-test-slave-7lbf krb5kdc[1675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:32.773502 1693 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:32.775358 1693 webserver.cc:460] Webserver started at http://127.0.0.1:45943/ using document root <none> and password file <none>
I20260805 22:52:32.776167 1693 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:32.776304 1693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:32.776633 1693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:32.780126 1693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "d540d07d5ce74d609828f737bb46f56c"
format_stamp: "Formatted at 2026-08-05 22:52:32 on dist-test-slave-7lbf"
server_key: "3571cd42917e7064ff9ceafb085a385c"
server_key_iv: "818b395fd3e1f7fe84b7d67998b9fa3d"
server_key_version: "encryptionkey@0"
I20260805 22:52:32.780997 1693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "d540d07d5ce74d609828f737bb46f56c"
format_stamp: "Formatted at 2026-08-05 22:52:32 on dist-test-slave-7lbf"
server_key: "3571cd42917e7064ff9ceafb085a385c"
server_key_iv: "818b395fd3e1f7fe84b7d67998b9fa3d"
server_key_version: "encryptionkey@0"
I20260805 22:52:32.787355 1693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260805 22:52:32.794281 1715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:32.799890 1693 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.004s
I20260805 22:52:32.800216 1693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "d540d07d5ce74d609828f737bb46f56c"
format_stamp: "Formatted at 2026-08-05 22:52:32 on dist-test-slave-7lbf"
server_key: "3571cd42917e7064ff9ceafb085a385c"
server_key_iv: "818b395fd3e1f7fe84b7d67998b9fa3d"
server_key_version: "encryptionkey@0"
I20260805 22:52:32.800575 1693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:32.827406 1693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:32.831981 1693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:32.832324 1693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:32.851548 1693 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43303
I20260805 22:52:32.851580 1788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43303 every 8 connection(s)
I20260805 22:52:32.853116 1693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:32.854423 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 1693
I20260805 22:52:32.854596 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:32.855027 4208 external_mini_cluster.cc:1428] Setting key 1f5be768bb545a4ed5b6c0d122701276
I20260805 22:52:32.858237 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:52:32 dist-test-slave-7lbf krb5kdc[1675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260805 22:52:32.872879 1790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c: Bootstrap starting.
I20260805 22:52:32.877449 1790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:32.878935 1790 log.cc:824] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:32.882979 1790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c: No bootstrap required, opened a new log
I20260805 22:52:32.883417 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:32.857119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52755 (local address 127.0.0.1:43303)
0805 22:52:32.858114 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:32.858132 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:32.858642 (+ 510us) server_negotiation.cc:407] Connection header received
0805 22:52:32.861124 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:32.861158 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:32.861838 (+ 680us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:32.862386 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:32.863919 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:32.866053 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:32.867660 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:32.868840 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:32.869078 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:32.873921 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:32.873933 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:32.873955 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:32.874021 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:32.878244 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:32.879115 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:32.879126 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:32.879136 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:32.879380 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:32.879968 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:32.879972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:32.879974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:32.880660 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:32.880973 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:32.881442 (+ 469us) server_negotiation.cc:299] Negotiation successful
0805 22:52:32.881972 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":437,"threads_started":1}
I20260805 22:52:32.887861 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:32.893252 1790 raft_consensus.cc:348] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d540d07d5ce74d609828f737bb46f56c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43303 } }
I20260805 22:52:32.893864 1790 raft_consensus.cc:374] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:32.894068 1790 raft_consensus.cc:729] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d540d07d5ce74d609828f737bb46f56c, State: Initialized, Role: FOLLOWER
I20260805 22:52:32.894882 1790 consensus_queue.cc:260] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d540d07d5ce74d609828f737bb46f56c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43303 } }
I20260805 22:52:32.895233 1790 raft_consensus.cc:388] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:32.895401 1790 raft_consensus.cc:482] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:32.895601 1790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:32.898685 1790 raft_consensus.cc:504] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d540d07d5ce74d609828f737bb46f56c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43303 } }
I20260805 22:52:32.899431 1790 leader_election.cc:302] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d540d07d5ce74d609828f737bb46f56c; no voters:
I20260805 22:52:32.900228 1790 leader_election.cc:288] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:32.900379 1811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:32.901975 1811 raft_consensus.cc:686] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [term 1 LEADER]: Becoming Leader. State: Replica: d540d07d5ce74d609828f737bb46f56c, State: Running, Role: LEADER
I20260805 22:52:32.902704 1811 consensus_queue.cc:237] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d540d07d5ce74d609828f737bb46f56c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43303 } }
I20260805 22:52:32.903033 1790 sys_catalog.cc:564] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:32.906165 1812 sys_catalog.cc:455] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d540d07d5ce74d609828f737bb46f56c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d540d07d5ce74d609828f737bb46f56c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43303 } } }
I20260805 22:52:32.906608 1812 sys_catalog.cc:458] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:32.906140 1813 sys_catalog.cc:455] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d540d07d5ce74d609828f737bb46f56c. Latest consensus state: current_term: 1 leader_uuid: "d540d07d5ce74d609828f737bb46f56c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d540d07d5ce74d609828f737bb46f56c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43303 } } }
I20260805 22:52:32.906733 1813 sys_catalog.cc:458] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:32.908212 1822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:32.912819 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:32.893747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44017 (local address 127.0.0.1:43303)
0805 22:52:32.893991 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:32.893999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:32.894633 (+ 634us) server_negotiation.cc:407] Connection header received
0805 22:52:32.894705 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:32.894709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:32.894785 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:32.894878 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:32.897089 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:32.897990 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:32.899781 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:32.900379 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:32.900525 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:32.903667 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:32.903673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:32.903676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:32.903717 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:32.907806 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:32.908953 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:32.908960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:32.908979 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:32.909269 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:32.909835 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:32.909841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:32.909845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:32.910315 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:32.910659 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:32.912138 (+ 1479us) server_negotiation.cc:299] Negotiation successful
0805 22:52:32.912412 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260805 22:52:32.918285 1822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:32.927872 1822 catalog_manager.cc:1347] Generated new cluster ID: 6dd1881d551f4bf18e0d5143e6534f60
I20260805 22:52:32.928028 1822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:32.961637 1822 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:32.963759 1822 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:32.981947 1822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d540d07d5ce74d609828f737bb46f56c: Generated new TSK 0
I20260805 22:52:32.982904 1822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:33.001235 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 1693
I20260805 22:52:33.002550 1783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:33.099911 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 1693
2026-08-05T22:52:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:33 dist-test-slave-7lbf krb5kdc[1838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:33 dist-test-slave-7lbf krb5kdc[1838](info): set up 2 sockets
Aug 05 22:52:33 dist-test-slave-7lbf krb5kdc[1838](info): commencing operation
krb5kdc: starting...
W20260805 22:52:34.346457 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:34 dist-test-slave-7lbf krb5kdc[1838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:34.720960 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:35.096150 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:35.096555 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:35.096691 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:35.104279 1857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:35.104421 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:35.104470 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:35.104532 1857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:35.104576 1857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:35.116196 1857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:35.117784 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:35.119252 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:35.132421 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:35.132344 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:35.132907 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:35.134281 1857 server_base.cc:1034] running on GCE node
I20260805 22:52:35.135288 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:35.137050 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:35.138345 1857 hybrid_clock.cc:648] HybridClock initialized: now 1785970355138299 us; error 35 us; skew 500 ppm
I20260805 22:52:35.138783 1857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:35 dist-test-slave-7lbf krb5kdc[1838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:35.142565 1857 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:35.144529 1857 webserver.cc:460] Webserver started at http://127.0.0.1:43095/ using document root <none> and password file <none>
I20260805 22:52:35.145366 1857 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:35.145505 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:35.145897 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:35.149430 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "d30edf1a6758443c9c1abbd56e6d4c39"
format_stamp: "Formatted at 2026-08-05 22:52:35 on dist-test-slave-7lbf"
server_key: "d88c748f44e0d773c3dfb0af7864c76c"
server_key_iv: "93cb933a98f43d632a74d14f0ca798b3"
server_key_version: "encryptionkey@0"
I20260805 22:52:35.150429 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "d30edf1a6758443c9c1abbd56e6d4c39"
format_stamp: "Formatted at 2026-08-05 22:52:35 on dist-test-slave-7lbf"
server_key: "d88c748f44e0d773c3dfb0af7864c76c"
server_key_iv: "93cb933a98f43d632a74d14f0ca798b3"
server_key_version: "encryptionkey@0"
I20260805 22:52:35.156701 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260805 22:52:35.161202 1881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:35.163026 1857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260805 22:52:35.163277 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "d30edf1a6758443c9c1abbd56e6d4c39"
format_stamp: "Formatted at 2026-08-05 22:52:35 on dist-test-slave-7lbf"
server_key: "d88c748f44e0d773c3dfb0af7864c76c"
server_key_iv: "93cb933a98f43d632a74d14f0ca798b3"
server_key_version: "encryptionkey@0"
I20260805 22:52:35.163516 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:35.200290 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:35.205487 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:35.205857 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:35.225908 1857 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37931
I20260805 22:52:35.225929 1949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37931 every 8 connection(s)
I20260805 22:52:35.227475 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:35.229420 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 1857
I20260805 22:52:35.229640 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:35.230094 4208 external_mini_cluster.cc:1428] Setting key f2a65ea56ecafd59e9f59a85524eed46
I20260805 22:52:35.232256 1953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:35.246304 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:35.232038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40803 (local address 127.0.0.1:37931)
0805 22:52:35.232914 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:35.232931 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:35.232985 (+ 54us) server_negotiation.cc:407] Connection header received
0805 22:52:35.235310 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:35.235343 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:35.235933 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:35.236436 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:35.237011 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:35.239150 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:35.240593 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:35.241679 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:35.241885 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:35.243052 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:35.243065 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:35.243086 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:35.243915 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:35.244037 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:35.244079 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:35.244196 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:35.244471 (+ 275us) server_negotiation.cc:299] Negotiation successful
0805 22:52:35.244803 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":346,"threads_started":1}
I20260805 22:52:35.247596 1953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39: Bootstrap starting.
I20260805 22:52:35.250855 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:35.251824 1953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:35.253263 1953 log.cc:824] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:35.257283 1953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39: No bootstrap required, opened a new log
I20260805 22:52:35.263633 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:35.256448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35809 (local address 127.0.0.1:37931)
0805 22:52:35.256657 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:35.256665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:35.257157 (+ 492us) server_negotiation.cc:407] Connection header received
0805 22:52:35.257329 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:35.257335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:35.257422 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:35.257572 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:35.258128 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:35.259277 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:35.260749 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:35.261480 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:35.261666 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:35.262781 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:35.262788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:35.262793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:35.262829 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:35.262893 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:35.262916 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:35.263011 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:35.263160 (+ 149us) server_negotiation.cc:299] Negotiation successful
0805 22:52:35.263270 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 22:52:35.267344 1953 raft_consensus.cc:348] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30edf1a6758443c9c1abbd56e6d4c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37931 } }
I20260805 22:52:35.267915 1953 raft_consensus.cc:374] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:35.268009 1953 raft_consensus.cc:729] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d30edf1a6758443c9c1abbd56e6d4c39, State: Initialized, Role: FOLLOWER
I20260805 22:52:35.268560 1953 consensus_queue.cc:260] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30edf1a6758443c9c1abbd56e6d4c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37931 } }
I20260805 22:52:35.268776 1953 raft_consensus.cc:388] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:35.268880 1953 raft_consensus.cc:482] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:35.269026 1953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:35.271375 1953 raft_consensus.cc:504] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30edf1a6758443c9c1abbd56e6d4c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37931 } }
I20260805 22:52:35.271910 1953 leader_election.cc:302] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d30edf1a6758443c9c1abbd56e6d4c39; no voters:
I20260805 22:52:35.272534 1953 leader_election.cc:288] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:35.272710 1966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:35.273976 1966 raft_consensus.cc:686] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [term 1 LEADER]: Becoming Leader. State: Replica: d30edf1a6758443c9c1abbd56e6d4c39, State: Running, Role: LEADER
I20260805 22:52:35.274637 1966 consensus_queue.cc:237] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30edf1a6758443c9c1abbd56e6d4c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37931 } }
I20260805 22:52:35.274920 1953 sys_catalog.cc:564] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:35.277878 1970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d30edf1a6758443c9c1abbd56e6d4c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30edf1a6758443c9c1abbd56e6d4c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37931 } } }
I20260805 22:52:35.277779 1974 sys_catalog.cc:455] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d30edf1a6758443c9c1abbd56e6d4c39. Latest consensus state: current_term: 1 leader_uuid: "d30edf1a6758443c9c1abbd56e6d4c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30edf1a6758443c9c1abbd56e6d4c39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37931 } } }
I20260805 22:52:35.278249 1974 sys_catalog.cc:458] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:35.278249 1970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:35.280331 1978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:35.286600 1978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:35.296159 1978 catalog_manager.cc:1347] Generated new cluster ID: 57210c1b87b44bfa80b9f999af959b41
I20260805 22:52:35.296298 1978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:35.317248 1978 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:35.318734 1978 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:35.328328 1978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d30edf1a6758443c9c1abbd56e6d4c39: Generated new TSK 0
I20260805 22:52:35.329263 1978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:35.335708 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 1857
W20260805 22:52:35.337172 1943 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40803
W20260805 22:52:35.338073 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 1857: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:35.338234 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 1857
2026-08-05T22:52:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:35 dist-test-slave-7lbf krb5kdc[1997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:35 dist-test-slave-7lbf krb5kdc[1997](info): set up 2 sockets
Aug 05 22:52:35 dist-test-slave-7lbf krb5kdc[1997](info): commencing operation
krb5kdc: starting...
W20260805 22:52:36.587775 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.002s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:36 dist-test-slave-7lbf krb5kdc[1997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:36.961951 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:37.336069 2014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:37.336404 2014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:37.336530 2014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:37.344157 2014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:37.344296 2014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:37.344352 2014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:37.344442 2014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:37.344494 2014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:37.354904 2014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:37.356362 2014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:37.357868 2014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:37.370610 2021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:37.370574 2024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:37.370515 2022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:37.371500 2014 server_base.cc:1034] running on GCE node
I20260805 22:52:37.372368 2014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:37.374111 2014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:37.375375 2014 hybrid_clock.cc:648] HybridClock initialized: now 1785970357375320 us; error 55 us; skew 500 ppm
I20260805 22:52:37.375792 2014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:37 dist-test-slave-7lbf krb5kdc[1997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:37.379382 2014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:37.381214 2014 webserver.cc:460] Webserver started at http://127.0.0.1:33785/ using document root <none> and password file <none>
I20260805 22:52:37.382042 2014 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:37.382182 2014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:37.382536 2014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:37.385831 2014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "67da5e6f1d5b427c92b97e715bf33c48"
format_stamp: "Formatted at 2026-08-05 22:52:37 on dist-test-slave-7lbf"
server_key: "a493ca63ae4832de3dfeca6c158a6058"
server_key_iv: "4a29c1fe84567df717fcda7f3baa1b51"
server_key_version: "encryptionkey@0"
I20260805 22:52:37.386679 2014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "67da5e6f1d5b427c92b97e715bf33c48"
format_stamp: "Formatted at 2026-08-05 22:52:37 on dist-test-slave-7lbf"
server_key: "a493ca63ae4832de3dfeca6c158a6058"
server_key_iv: "4a29c1fe84567df717fcda7f3baa1b51"
server_key_version: "encryptionkey@0"
I20260805 22:52:37.392794 2014 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260805 22:52:37.397202 2039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:37.398902 2014 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:52:37.399104 2014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "67da5e6f1d5b427c92b97e715bf33c48"
format_stamp: "Formatted at 2026-08-05 22:52:37 on dist-test-slave-7lbf"
server_key: "a493ca63ae4832de3dfeca6c158a6058"
server_key_iv: "4a29c1fe84567df717fcda7f3baa1b51"
server_key_version: "encryptionkey@0"
I20260805 22:52:37.399323 2014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:37.405988 2014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:37.410887 2014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:37.411229 2014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:37.430660 2014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42037
I20260805 22:52:37.430696 2116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42037 every 8 connection(s)
I20260805 22:52:37.432260 2014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:37.436446 2117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:37.439196 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 2014
I20260805 22:52:37.439399 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:37.439961 4208 external_mini_cluster.cc:1428] Setting key 8eb9e049846218f417d4e0463fa04a72
I20260805 22:52:37.448949 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:37.442230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59099 (local address 127.0.0.1:42037)
0805 22:52:37.443193 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:37.443207 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:37.443257 (+ 50us) server_negotiation.cc:407] Connection header received
0805 22:52:37.445067 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:37.445098 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:37.445503 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:37.446001 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:37.447159 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:37.447166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:37.447180 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:37.447200 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:37.447272 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:37.447297 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:37.447574 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:37.447720 (+ 146us) server_negotiation.cc:299] Negotiation successful
0805 22:52:37.447930 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":416,"threads_started":1}
I20260805 22:52:37.449857 2117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48: Bootstrap starting.
I20260805 22:52:37.453143 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:37.453465 2117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:37.454644 2117 log.cc:824] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:37.458421 2117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48: No bootstrap required, opened a new log
I20260805 22:52:37.460407 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:37.457942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44393 (local address 127.0.0.1:42037)
0805 22:52:37.458152 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:37.458158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:37.458469 (+ 311us) server_negotiation.cc:407] Connection header received
0805 22:52:37.458658 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:37.458662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:37.458719 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:37.458805 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:37.459710 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:37.459716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:37.459720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:37.459737 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:37.459781 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:37.459797 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:37.459906 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:37.460049 (+ 143us) server_negotiation.cc:299] Negotiation successful
0805 22:52:37.460118 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260805 22:52:37.467370 2117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da5e6f1d5b427c92b97e715bf33c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260805 22:52:37.467864 2117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:37.467970 2117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67da5e6f1d5b427c92b97e715bf33c48, State: Initialized, Role: FOLLOWER
I20260805 22:52:37.468636 2117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da5e6f1d5b427c92b97e715bf33c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260805 22:52:37.468918 2117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:37.469101 2117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:37.469339 2117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:37.471963 2117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da5e6f1d5b427c92b97e715bf33c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260805 22:52:37.472568 2117 leader_election.cc:302] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67da5e6f1d5b427c92b97e715bf33c48; no voters:
I20260805 22:52:37.473197 2117 leader_election.cc:288] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:37.473371 2131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:37.474061 2131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [term 1 LEADER]: Becoming Leader. State: Replica: 67da5e6f1d5b427c92b97e715bf33c48, State: Running, Role: LEADER
I20260805 22:52:37.474750 2131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da5e6f1d5b427c92b97e715bf33c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260805 22:52:37.475451 2117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:37.478016 2133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67da5e6f1d5b427c92b97e715bf33c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da5e6f1d5b427c92b97e715bf33c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } } }
I20260805 22:52:37.478488 2133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:37.478868 2134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67da5e6f1d5b427c92b97e715bf33c48. Latest consensus state: current_term: 1 leader_uuid: "67da5e6f1d5b427c92b97e715bf33c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da5e6f1d5b427c92b97e715bf33c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } } }
I20260805 22:52:37.479157 2134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:37.480417 2144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:37.485458 2144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:37.494791 2144 catalog_manager.cc:1347] Generated new cluster ID: 5cd1ffc39cd24bbea39ed0f74d831271
I20260805 22:52:37.494936 2144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:37.506388 2144 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:37.506992 2144 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:37.514526 2144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67da5e6f1d5b427c92b97e715bf33c48: Generated new TSK 0
I20260805 22:52:37.515460 2144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:37.528311 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2014
W20260805 22:52:37.529629 2112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59099
W20260805 22:52:37.530508 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2014: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:37.530716 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 2014
2026-08-05T22:52:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:37 dist-test-slave-7lbf krb5kdc[2160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:37 dist-test-slave-7lbf krb5kdc[2160](info): set up 2 sockets
Aug 05 22:52:37 dist-test-slave-7lbf krb5kdc[2160](info): commencing operation
krb5kdc: starting...
W20260805 22:52:38.789674 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:39 dist-test-slave-7lbf krb5kdc[2160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:39.164815 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:39.541991 2179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:39.542338 2179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:39.542470 2179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:39.550209 2179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:39.550359 2179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:39.550419 2179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:39.550508 2179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:39.550581 2179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:39.561142 2179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:39.562772 2179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:39.564263 2179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:39.576889 2188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:39.576910 2189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:39.576812 2194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:39.578151 2179 server_base.cc:1034] running on GCE node
I20260805 22:52:39.579051 2179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:39.580716 2179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:39.581952 2179 hybrid_clock.cc:648] HybridClock initialized: now 1785970359581890 us; error 56 us; skew 500 ppm
I20260805 22:52:39.582355 2179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:39 dist-test-slave-7lbf krb5kdc[2160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:39.585877 2179 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:39.587628 2179 webserver.cc:460] Webserver started at http://127.0.0.1:43333/ using document root <none> and password file <none>
I20260805 22:52:39.588435 2179 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:39.588568 2179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:39.588907 2179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:39.592370 2179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "e30dab8976c2492481ff10bb51f75ac2"
format_stamp: "Formatted at 2026-08-05 22:52:39 on dist-test-slave-7lbf"
server_key: "98ba0105f05c3457c84c80b93d4c58ce"
server_key_iv: "58bb2d6a2b588ec3f7530127ccf6c30b"
server_key_version: "encryptionkey@0"
I20260805 22:52:39.593289 2179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "e30dab8976c2492481ff10bb51f75ac2"
format_stamp: "Formatted at 2026-08-05 22:52:39 on dist-test-slave-7lbf"
server_key: "98ba0105f05c3457c84c80b93d4c58ce"
server_key_iv: "58bb2d6a2b588ec3f7530127ccf6c30b"
server_key_version: "encryptionkey@0"
I20260805 22:52:39.599408 2179 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260805 22:52:39.603564 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:39.605242 2179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 22:52:39.605465 2179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "e30dab8976c2492481ff10bb51f75ac2"
format_stamp: "Formatted at 2026-08-05 22:52:39 on dist-test-slave-7lbf"
server_key: "98ba0105f05c3457c84c80b93d4c58ce"
server_key_iv: "58bb2d6a2b588ec3f7530127ccf6c30b"
server_key_version: "encryptionkey@0"
I20260805 22:52:39.605762 2179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:39.645033 2179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:39.649893 2179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:39.650198 2179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:39.669549 2179 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40245
I20260805 22:52:39.669644 2271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40245 every 8 connection(s)
I20260805 22:52:39.671119 2179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:39.673951 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 2179
I20260805 22:52:39.674149 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:39.674608 4208 external_mini_cluster.cc:1428] Setting key b2902b2fda761e7de266aa93176672e4
I20260805 22:52:39.675521 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:52:39 dist-test-slave-7lbf krb5kdc[2160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260805 22:52:39.691152 2272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2: Bootstrap starting.
I20260805 22:52:39.695111 2272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:39.696576 2272 log.cc:824] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:39.700275 2272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2: No bootstrap required, opened a new log
I20260805 22:52:39.701014 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:39.676761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:37155 (local address 127.0.0.1:40245)
0805 22:52:39.677642 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:39.677656 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:39.677704 (+ 48us) server_negotiation.cc:407] Connection header received
0805 22:52:39.679406 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:39.679433 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:39.679897 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:39.680297 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:39.682275 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:39.683825 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:39.685731 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:39.686474 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:39.686623 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:39.692106 (+ 5483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:39.692118 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:39.692139 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:39.692188 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:39.695559 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:39.696840 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:39.696852 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:39.696860 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:39.697121 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:39.697752 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:39.697758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:39.697761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:39.698470 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:39.698742 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:39.699396 (+ 654us) server_negotiation.cc:299] Negotiation successful
0805 22:52:39.699794 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":336,"threads_started":1}
I20260805 22:52:39.705858 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:39.709249 2272 raft_consensus.cc:348] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30dab8976c2492481ff10bb51f75ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } }
I20260805 22:52:39.709775 2272 raft_consensus.cc:374] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:39.709888 2272 raft_consensus.cc:729] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30dab8976c2492481ff10bb51f75ac2, State: Initialized, Role: FOLLOWER
I20260805 22:52:39.710582 2272 consensus_queue.cc:260] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30dab8976c2492481ff10bb51f75ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } }
I20260805 22:52:39.710927 2272 raft_consensus.cc:388] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:39.711146 2272 raft_consensus.cc:482] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:39.711386 2272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:39.714792 2272 raft_consensus.cc:504] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30dab8976c2492481ff10bb51f75ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } }
I20260805 22:52:39.715559 2272 leader_election.cc:302] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e30dab8976c2492481ff10bb51f75ac2; no voters:
I20260805 22:52:39.716368 2272 leader_election.cc:288] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:39.716482 2287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:39.719108 2287 raft_consensus.cc:686] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [term 1 LEADER]: Becoming Leader. State: Replica: e30dab8976c2492481ff10bb51f75ac2, State: Running, Role: LEADER
I20260805 22:52:39.719066 2272 sys_catalog.cc:564] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:39.719769 2287 consensus_queue.cc:237] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30dab8976c2492481ff10bb51f75ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } }
I20260805 22:52:39.723294 2290 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e30dab8976c2492481ff10bb51f75ac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30dab8976c2492481ff10bb51f75ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } } }
I20260805 22:52:39.723881 2290 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:39.723698 2292 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30dab8976c2492481ff10bb51f75ac2. Latest consensus state: current_term: 1 leader_uuid: "e30dab8976c2492481ff10bb51f75ac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30dab8976c2492481ff10bb51f75ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } } }
I20260805 22:52:39.724048 2292 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:39.732544 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:39.710806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:34903 (local address 127.0.0.1:40245)
0805 22:52:39.711102 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:39.711109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:39.711752 (+ 643us) server_negotiation.cc:407] Connection header received
0805 22:52:39.711973 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:39.711977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:39.712053 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:39.712153 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:39.714227 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:39.715067 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:39.717048 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:39.717735 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:39.717862 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:39.722081 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:39.722090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:39.722098 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:39.722156 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:39.726767 (+ 4611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:39.728213 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:39.728221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:39.728226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:39.728535 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:39.729271 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:39.729278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:39.729282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:39.729789 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:39.730144 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:39.731853 (+ 1709us) server_negotiation.cc:299] Negotiation successful
0805 22:52:39.732109 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260805 22:52:39.736778 2299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:39.741854 2299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:39.752542 2299 catalog_manager.cc:1347] Generated new cluster ID: bf3ce85cbb864b30b996ddff90d6a72d
I20260805 22:52:39.752876 2299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:39.773310 2299 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:39.774796 2299 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:39.789409 2299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e30dab8976c2492481ff10bb51f75ac2: Generated new TSK 0
I20260805 22:52:39.790740 2299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:39.804121 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2179
I20260805 22:52:39.805739 2263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:39.901355 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 2179
W20260805 22:52:39.928596 2158 connection.cc:535] client connection to 127.0.0.1:40245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40245): Connection reset by peer (error 104)
2026-08-05T22:52:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:40 dist-test-slave-7lbf krb5kdc[2315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:40 dist-test-slave-7lbf krb5kdc[2315](info): set up 2 sockets
Aug 05 22:52:40 dist-test-slave-7lbf krb5kdc[2315](info): commencing operation
krb5kdc: starting...
W20260805 22:52:41.155441 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.009s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:41 dist-test-slave-7lbf krb5kdc[2315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:41.524230 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:41.890604 2332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:41.890946 2332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:41.891088 2332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:41.898533 2332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:41.898680 2332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:41.898759 2332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:41.898849 2332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:41.898910 2332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:41.909250 2332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:41.910796 2332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:41.912218 2332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:41.925024 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:41.924989 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:41.924950 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:41.925992 2332 server_base.cc:1034] running on GCE node
I20260805 22:52:41.926887 2332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:41.928553 2332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:41.929774 2332 hybrid_clock.cc:648] HybridClock initialized: now 1785970361929717 us; error 54 us; skew 500 ppm
I20260805 22:52:41.930202 2332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:41 dist-test-slave-7lbf krb5kdc[2315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:41.933621 2332 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:41.935254 2332 webserver.cc:460] Webserver started at http://127.0.0.1:40591/ using document root <none> and password file <none>
I20260805 22:52:41.936002 2332 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:41.936131 2332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:41.936450 2332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:41.939723 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "f7089a6a7ddd4534971477d91ecb3ba7"
format_stamp: "Formatted at 2026-08-05 22:52:41 on dist-test-slave-7lbf"
server_key: "d854b6a21256b5712ab8a51a2b910f8e"
server_key_iv: "3c7d9d0c6453a7783432d6f28207e611"
server_key_version: "encryptionkey@0"
I20260805 22:52:41.940600 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "f7089a6a7ddd4534971477d91ecb3ba7"
format_stamp: "Formatted at 2026-08-05 22:52:41 on dist-test-slave-7lbf"
server_key: "d854b6a21256b5712ab8a51a2b910f8e"
server_key_iv: "3c7d9d0c6453a7783432d6f28207e611"
server_key_version: "encryptionkey@0"
I20260805 22:52:41.946476 2332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260805 22:52:41.950273 2352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:41.951740 2332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 22:52:41.951920 2332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "f7089a6a7ddd4534971477d91ecb3ba7"
format_stamp: "Formatted at 2026-08-05 22:52:41 on dist-test-slave-7lbf"
server_key: "d854b6a21256b5712ab8a51a2b910f8e"
server_key_iv: "3c7d9d0c6453a7783432d6f28207e611"
server_key_version: "encryptionkey@0"
I20260805 22:52:41.952121 2332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:41.968976 2332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:41.974026 2332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:41.974334 2332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:41.994020 2332 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44109
I20260805 22:52:41.994055 2420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44109 every 8 connection(s)
I20260805 22:52:41.995503 2332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:41.999616 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:42.001856 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 2332
I20260805 22:52:42.002064 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:42.002643 4208 external_mini_cluster.cc:1428] Setting key f27e9c88387c9f5b00928f3001bb25a4
I20260805 22:52:42.014118 2423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7: Bootstrap starting.
I20260805 22:52:42.016855 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:42.004795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:43019 (local address 127.0.0.1:44109)
0805 22:52:42.005765 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:42.005779 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:42.005839 (+ 60us) server_negotiation.cc:407] Connection header received
0805 22:52:42.007550 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:42.007583 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:42.008077 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:42.008502 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:42.009123 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:42.010563 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:42.012051 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:42.012830 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:42.013017 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:42.014420 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:42.014430 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:42.014448 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:42.015049 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:42.015120 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:42.015154 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:42.015316 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:42.015640 (+ 324us) server_negotiation.cc:299] Negotiation successful
0805 22:52:42.015830 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":399,"threads_started":1}
I20260805 22:52:42.017714 2423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:42.018831 2423 log.cc:824] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:42.021018 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:42.022042 2423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7: No bootstrap required, opened a new log
I20260805 22:52:42.031733 2423 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7089a6a7ddd4534971477d91ecb3ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44109 } }
I20260805 22:52:42.032275 2423 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:42.032420 2423 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7089a6a7ddd4534971477d91ecb3ba7, State: Initialized, Role: FOLLOWER
I20260805 22:52:42.032975 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:42.025785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:54697 (local address 127.0.0.1:44109)
0805 22:52:42.026020 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:42.026027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:42.026362 (+ 335us) server_negotiation.cc:407] Connection header received
0805 22:52:42.026518 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:42.026523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:42.026620 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:42.026756 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:42.027344 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:42.028544 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:42.030059 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:42.030782 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:42.030915 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:42.032107 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:42.032114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:42.032119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:42.032145 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:42.032197 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:42.032222 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:42.032348 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:42.032565 (+ 217us) server_negotiation.cc:299] Negotiation successful
0805 22:52:42.032627 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260805 22:52:42.033219 2423 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7089a6a7ddd4534971477d91ecb3ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44109 } }
I20260805 22:52:42.033502 2423 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:42.033747 2423 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:42.033967 2423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:42.036568 2423 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7089a6a7ddd4534971477d91ecb3ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44109 } }
I20260805 22:52:42.037093 2423 leader_election.cc:302] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7089a6a7ddd4534971477d91ecb3ba7; no voters:
I20260805 22:52:42.037732 2423 leader_election.cc:288] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:42.037884 2438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:42.038620 2438 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [term 1 LEADER]: Becoming Leader. State: Replica: f7089a6a7ddd4534971477d91ecb3ba7, State: Running, Role: LEADER
I20260805 22:52:42.039299 2438 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7089a6a7ddd4534971477d91ecb3ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44109 } }
I20260805 22:52:42.039999 2423 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:42.042105 2441 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7089a6a7ddd4534971477d91ecb3ba7. Latest consensus state: current_term: 1 leader_uuid: "f7089a6a7ddd4534971477d91ecb3ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7089a6a7ddd4534971477d91ecb3ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44109 } } }
I20260805 22:52:42.042424 2441 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:42.043296 2447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:42.043154 2440 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7089a6a7ddd4534971477d91ecb3ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7089a6a7ddd4534971477d91ecb3ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44109 } } }
I20260805 22:52:42.048664 2440 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:42.053009 2447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:42.066306 2447 catalog_manager.cc:1347] Generated new cluster ID: 8a216cbb2768470d99164c47ce78b2cf
I20260805 22:52:42.066461 2447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:42.084460 2447 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:42.085914 2447 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:42.097211 2447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7089a6a7ddd4534971477d91ecb3ba7: Generated new TSK 0
I20260805 22:52:42.098244 2447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:42.113822 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2332
W20260805 22:52:42.115592 2413 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.224:43019
W20260805 22:52:42.116602 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2332: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:42.116797 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 2332
W20260805 22:52:42.141940 2313 connection.cc:535] client connection to 127.0.0.1:44109 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44109): Connection reset by peer (error 104)
2026-08-05T22:52:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:42 dist-test-slave-7lbf krb5kdc[2467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:42 dist-test-slave-7lbf krb5kdc[2467](info): set up 2 sockets
Aug 05 22:52:42 dist-test-slave-7lbf krb5kdc[2467](info): commencing operation
krb5kdc: starting...
W20260805 22:52:43.372905 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:43 dist-test-slave-7lbf krb5kdc[2467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:43.740964 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:44.113209 2481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:44.113617 2481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:44.113741 2481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:44.121387 2481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:44.121578 2481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:44.121650 2481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:44.121723 2481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:44.121769 2481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:44.132484 2481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:44.134073 2481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:44.135535 2481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:44.148792 2489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:44.148819 2492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:44.149008 2490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:44.149617 2481 server_base.cc:1034] running on GCE node
I20260805 22:52:44.150606 2481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:44.152467 2481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:44.153728 2481 hybrid_clock.cc:648] HybridClock initialized: now 1785970364153673 us; error 46 us; skew 500 ppm
I20260805 22:52:44.154161 2481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:44 dist-test-slave-7lbf krb5kdc[2467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:44.161319 2481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:44.164005 2481 webserver.cc:460] Webserver started at http://127.0.0.1:38135/ using document root <none> and password file <none>
I20260805 22:52:44.165161 2481 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:44.165354 2481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:44.165819 2481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:44.169296 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "7776761c94934a4aa6f8b1aadc6d48c5"
format_stamp: "Formatted at 2026-08-05 22:52:44 on dist-test-slave-7lbf"
server_key: "5a8ea503fb5f2199b832359edbc35534"
server_key_iv: "c58e865fef4e7b58759efa4ffcc306d3"
server_key_version: "encryptionkey@0"
I20260805 22:52:44.170289 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "7776761c94934a4aa6f8b1aadc6d48c5"
format_stamp: "Formatted at 2026-08-05 22:52:44 on dist-test-slave-7lbf"
server_key: "5a8ea503fb5f2199b832359edbc35534"
server_key_iv: "c58e865fef4e7b58759efa4ffcc306d3"
server_key_version: "encryptionkey@0"
I20260805 22:52:44.176558 2481 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260805 22:52:44.181157 2504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:44.182915 2481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:52:44.183158 2481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "7776761c94934a4aa6f8b1aadc6d48c5"
format_stamp: "Formatted at 2026-08-05 22:52:44 on dist-test-slave-7lbf"
server_key: "5a8ea503fb5f2199b832359edbc35534"
server_key_iv: "c58e865fef4e7b58759efa4ffcc306d3"
server_key_version: "encryptionkey@0"
I20260805 22:52:44.183388 2481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:44.189963 2481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:44.194994 2481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:44.195327 2481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:44.214452 2481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43621
I20260805 22:52:44.214502 2573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43621 every 8 connection(s)
I20260805 22:52:44.216013 2481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:44.218394 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 2481
I20260805 22:52:44.218601 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:44.219203 4208 external_mini_cluster.cc:1428] Setting key 70a48f29d1750bb392181fb4f1e97f1e
I20260805 22:52:44.221057 2574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:44.228256 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:44.221457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:59643 (local address 127.0.0.1:43621)
0805 22:52:44.222393 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:44.222410 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:44.222467 (+ 57us) server_negotiation.cc:407] Connection header received
0805 22:52:44.224233 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:44.224262 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:44.224694 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:44.225155 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:44.226348 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:44.226356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:44.226377 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:44.226397 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:44.226483 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:44.226509 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:44.226745 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:44.226931 (+ 186us) server_negotiation.cc:299] Negotiation successful
0805 22:52:44.227159 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":377,"threads_started":1}
I20260805 22:52:44.232625 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:44.234637 2574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5: Bootstrap starting.
I20260805 22:52:44.238548 2574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:44.239817 2574 log.cc:824] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:44.240126 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:44.237394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:58919 (local address 127.0.0.1:43621)
0805 22:52:44.237612 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:44.237623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:44.238034 (+ 411us) server_negotiation.cc:407] Connection header received
0805 22:52:44.238215 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:44.238219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:44.238274 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:44.238358 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:44.239371 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:44.239377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:44.239382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:44.239401 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:44.239444 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:44.239460 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:44.239569 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:44.239716 (+ 147us) server_negotiation.cc:299] Negotiation successful
0805 22:52:44.239781 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 22:52:44.243292 2574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5: No bootstrap required, opened a new log
I20260805 22:52:44.251576 2574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7776761c94934a4aa6f8b1aadc6d48c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } }
I20260805 22:52:44.252187 2574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:44.252311 2574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7776761c94934a4aa6f8b1aadc6d48c5, State: Initialized, Role: FOLLOWER
I20260805 22:52:44.253050 2574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7776761c94934a4aa6f8b1aadc6d48c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } }
I20260805 22:52:44.253304 2574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:44.253453 2574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:44.253650 2574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:44.256124 2574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7776761c94934a4aa6f8b1aadc6d48c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } }
I20260805 22:52:44.256734 2574 leader_election.cc:302] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7776761c94934a4aa6f8b1aadc6d48c5; no voters:
I20260805 22:52:44.257405 2574 leader_election.cc:288] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:44.257619 2587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:44.258327 2587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [term 1 LEADER]: Becoming Leader. State: Replica: 7776761c94934a4aa6f8b1aadc6d48c5, State: Running, Role: LEADER
I20260805 22:52:44.258956 2587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7776761c94934a4aa6f8b1aadc6d48c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } }
I20260805 22:52:44.259807 2574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:44.262253 2589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7776761c94934a4aa6f8b1aadc6d48c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7776761c94934a4aa6f8b1aadc6d48c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } } }
I20260805 22:52:44.262377 2590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7776761c94934a4aa6f8b1aadc6d48c5. Latest consensus state: current_term: 1 leader_uuid: "7776761c94934a4aa6f8b1aadc6d48c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7776761c94934a4aa6f8b1aadc6d48c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } } }
I20260805 22:52:44.263049 2589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:44.263078 2590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:44.265215 2600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:44.270852 2600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:44.280956 2600 catalog_manager.cc:1347] Generated new cluster ID: 56e9989f17274dcea018be63e4850f8b
I20260805 22:52:44.281087 2600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:44.306048 2600 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:44.306555 2600 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:44.316423 2600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7776761c94934a4aa6f8b1aadc6d48c5: Generated new TSK 0
I20260805 22:52:44.317348 2600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:44.324916 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2481
W20260805 22:52:44.326193 2568 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.224:59643
W20260805 22:52:44.326967 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2481: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:44.327113 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 2481
2026-08-05T22:52:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:44 dist-test-slave-7lbf krb5kdc[2618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:44 dist-test-slave-7lbf krb5kdc[2618](info): set up 2 sockets
Aug 05 22:52:44 dist-test-slave-7lbf krb5kdc[2618](info): commencing operation
krb5kdc: starting...
W20260805 22:52:45.585716 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:45 dist-test-slave-7lbf krb5kdc[2618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:45.956802 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:46.334126 2632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:46.334462 2632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:46.334607 2632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:46.342322 2632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:46.342463 2632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:46.342530 2632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:46.342623 2632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:46.342684 2632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:46.353230 2632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37375
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:46.354740 2632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:46.356153 2632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:46.368708 2642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:46.368762 2643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:46.368793 2645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:46.369830 2632 server_base.cc:1034] running on GCE node
I20260805 22:52:46.370710 2632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:46.372249 2632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:46.373474 2632 hybrid_clock.cc:648] HybridClock initialized: now 1785970366373422 us; error 46 us; skew 500 ppm
I20260805 22:52:46.373898 2632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:46 dist-test-slave-7lbf krb5kdc[2618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:46.377475 2632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:46.379423 2632 webserver.cc:460] Webserver started at http://127.0.0.1:44613/ using document root <none> and password file <none>
I20260805 22:52:46.380290 2632 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:46.380434 2632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:46.380774 2632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:46.384197 2632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43"
format_stamp: "Formatted at 2026-08-05 22:52:46 on dist-test-slave-7lbf"
server_key: "3ccea3775090802c56b43848899cec12"
server_key_iv: "1b8464ab830e0db0e58ced89b96c2052"
server_key_version: "encryptionkey@0"
I20260805 22:52:46.385113 2632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43"
format_stamp: "Formatted at 2026-08-05 22:52:46 on dist-test-slave-7lbf"
server_key: "3ccea3775090802c56b43848899cec12"
server_key_iv: "1b8464ab830e0db0e58ced89b96c2052"
server_key_version: "encryptionkey@0"
I20260805 22:52:46.391425 2632 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260805 22:52:46.395957 2655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:46.397722 2632 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:52:46.397971 2632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43"
format_stamp: "Formatted at 2026-08-05 22:52:46 on dist-test-slave-7lbf"
server_key: "3ccea3775090802c56b43848899cec12"
server_key_iv: "1b8464ab830e0db0e58ced89b96c2052"
server_key_version: "encryptionkey@0"
I20260805 22:52:46.398224 2632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:46.447424 2632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:46.452212 2632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:46.452524 2632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:46.472008 2632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37375
I20260805 22:52:46.472028 2718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37375 every 8 connection(s)
I20260805 22:52:46.473500 2632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:46.475674 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 2632
I20260805 22:52:46.475880 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:46.476493 4208 external_mini_cluster.cc:1428] Setting key 16e4895d7abaaa067c9e1262a3b6c638
I20260805 22:52:46.478694 2719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 05 22:52:46 dist-test-slave-7lbf krb5kdc[2618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260805 22:52:46.493000 2719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43: Bootstrap starting.
I20260805 22:52:46.496644 2719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:46.498111 2719 log.cc:824] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:46.501936 2719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43: No bootstrap required, opened a new log
I20260805 22:52:46.502419 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:46.478499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:55743 (local address 127.0.0.1:37375)
0805 22:52:46.479375 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:46.479388 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:46.479435 (+ 47us) server_negotiation.cc:407] Connection header received
0805 22:52:46.481171 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:46.481197 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:46.481719 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:46.482100 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:46.484088 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:46.485637 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:46.487453 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:46.488218 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:46.488381 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:46.493807 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:46.493818 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:46.493837 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:46.493885 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:46.497079 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:46.498275 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:46.498285 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:46.498296 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:46.498492 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:46.499126 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:46.499132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:46.499135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:46.499859 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:46.500136 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:46.500794 (+ 658us) server_negotiation.cc:299] Negotiation successful
0805 22:52:46.501136 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":406,"threads_started":1}
I20260805 22:52:46.506924 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:46.510406 2719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } }
I20260805 22:52:46.510851 2719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:46.510969 2719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd9eeb6e56e42d2a7963e0986c6ef43, State: Initialized, Role: FOLLOWER
I20260805 22:52:46.511734 2719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } }
I20260805 22:52:46.512042 2719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:46.512185 2719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:46.512362 2719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:46.514833 2719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } }
I20260805 22:52:46.515396 2719 leader_election.cc:302] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fd9eeb6e56e42d2a7963e0986c6ef43; no voters:
I20260805 22:52:46.516054 2719 leader_election.cc:288] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:46.516176 2737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:46.517391 2737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [term 1 LEADER]: Becoming Leader. State: Replica: 0fd9eeb6e56e42d2a7963e0986c6ef43, State: Running, Role: LEADER
I20260805 22:52:46.518083 2737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } }
I20260805 22:52:46.518678 2719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:46.521621 2743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fd9eeb6e56e42d2a7963e0986c6ef43. Latest consensus state: current_term: 1 leader_uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } } }
I20260805 22:52:46.521690 2738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fd9eeb6e56e42d2a7963e0986c6ef43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } } }
I20260805 22:52:46.522137 2743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:46.522137 2738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:46.529886 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:46.511331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:60929 (local address 127.0.0.1:37375)
0805 22:52:46.511524 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:46.511531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:46.512024 (+ 493us) server_negotiation.cc:407] Connection header received
0805 22:52:46.512251 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:46.512257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:46.512363 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:46.512449 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:46.514075 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:46.514890 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:46.516280 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:46.517035 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:46.517914 (+ 879us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:46.520128 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:46.520136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:46.520140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:46.520184 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:46.524338 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:46.525798 (+ 1460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:46.525805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:46.525809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:46.526008 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:46.526726 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:46.526732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:46.526736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:46.527238 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:46.528891 (+ 1653us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:46.529118 (+ 227us) server_negotiation.cc:299] Negotiation successful
0805 22:52:46.529401 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 22:52:46.535104 2747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:46.541080 2747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:46.551805 2747 catalog_manager.cc:1347] Generated new cluster ID: 8250bd3820904b099d624828d3871deb
I20260805 22:52:46.551954 2747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:46.587018 2747 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:46.588621 2747 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:46.606170 2747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fd9eeb6e56e42d2a7963e0986c6ef43: Generated new TSK 0
I20260805 22:52:46.607141 2747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:46.623096 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2632
I20260805 22:52:46.624969 2714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:46.722716 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 2632
W20260805 22:52:46.750188 2613 connection.cc:535] client connection to 127.0.0.1:37375 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37375): Connection reset by peer (error 104)
2026-08-05T22:52:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:46 dist-test-slave-7lbf krb5kdc[2776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:46 dist-test-slave-7lbf krb5kdc[2776](info): set up 2 sockets
Aug 05 22:52:46 dist-test-slave-7lbf krb5kdc[2776](info): commencing operation
krb5kdc: starting...
W20260805 22:52:47.972090 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.003s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:48 dist-test-slave-7lbf krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:48.349431 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:48.721822 2796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:48.722180 2796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:48.722326 2796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:48.731056 2796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:48.731257 2796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:48.731371 2796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:48.731454 2796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:48.731520 2796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:48.745024 2796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:48.746968 2796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:48.748615 2796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:48.763512 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:48.763823 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:48.763837 2806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:48.767165 2796 server_base.cc:1034] running on GCE node
I20260805 22:52:48.768440 2796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:48.770704 2796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:48.772038 2796 hybrid_clock.cc:648] HybridClock initialized: now 1785970368771979 us; error 43 us; skew 500 ppm
I20260805 22:52:48.772532 2796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:48 dist-test-slave-7lbf krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:48.777284 2796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:48.779289 2796 webserver.cc:460] Webserver started at http://127.0.0.1:39157/ using document root <none> and password file <none>
I20260805 22:52:48.780072 2796 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:48.780220 2796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:48.780563 2796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:48.784044 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "729d6d6cd3a745d281dcfa78f6fd8dea"
format_stamp: "Formatted at 2026-08-05 22:52:48 on dist-test-slave-7lbf"
server_key: "ab428352372d5692a551f08d98928c97"
server_key_iv: "7295fa7e8b449cc6a5a57b08f36026b8"
server_key_version: "encryptionkey@0"
I20260805 22:52:48.784960 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "729d6d6cd3a745d281dcfa78f6fd8dea"
format_stamp: "Formatted at 2026-08-05 22:52:48 on dist-test-slave-7lbf"
server_key: "ab428352372d5692a551f08d98928c97"
server_key_iv: "7295fa7e8b449cc6a5a57b08f36026b8"
server_key_version: "encryptionkey@0"
I20260805 22:52:48.792160 2796 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260805 22:52:48.796607 2813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:48.798300 2796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:52:48.798507 2796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "729d6d6cd3a745d281dcfa78f6fd8dea"
format_stamp: "Formatted at 2026-08-05 22:52:48 on dist-test-slave-7lbf"
server_key: "ab428352372d5692a551f08d98928c97"
server_key_iv: "7295fa7e8b449cc6a5a57b08f36026b8"
server_key_version: "encryptionkey@0"
I20260805 22:52:48.798728 2796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:48.811730 2796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:48.816437 2796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:48.816745 2796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:48.835865 2796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34785
I20260805 22:52:48.835963 2876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34785 every 8 connection(s)
I20260805 22:52:48.837435 2796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:48.841879 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:48.842374 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 2796
I20260805 22:52:48.842566 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:48.843122 4208 external_mini_cluster.cc:1428] Setting key 8168a9781d077cb88f7bdaa7b2b8a6bd
I20260805 22:52:48.857247 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea: Bootstrap starting.
I20260805 22:52:48.858991 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:48.845186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:56205 (local address 127.0.0.1:34785)
0805 22:52:48.846110 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:48.846125 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:48.846172 (+ 47us) server_negotiation.cc:407] Connection header received
0805 22:52:48.848575 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:48.848612 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:48.849130 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:48.849564 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:48.850262 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:48.851638 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:48.853854 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:48.854683 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:48.854840 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:48.856268 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:48.856280 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:48.856298 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:48.856948 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:48.857042 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:48.857078 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:48.857211 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:48.857653 (+ 442us) server_negotiation.cc:299] Negotiation successful
0805 22:52:48.857867 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":428,"threads_started":1}
I20260805 22:52:48.861208 2878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:48.862576 2878 log.cc:824] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:48.863705 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:48.865893 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea: No bootstrap required, opened a new log
I20260805 22:52:48.874228 2878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d6d6cd3a745d281dcfa78f6fd8dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34785 } }
I20260805 22:52:48.874779 2878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:48.874895 2878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729d6d6cd3a745d281dcfa78f6fd8dea, State: Initialized, Role: FOLLOWER
I20260805 22:52:48.875551 2878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d6d6cd3a745d281dcfa78f6fd8dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34785 } }
I20260805 22:52:48.875816 2878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:48.875965 2878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:48.876152 2878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:48.879253 2878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d6d6cd3a745d281dcfa78f6fd8dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34785 } }
I20260805 22:52:48.879606 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:48.871714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:56653 (local address 127.0.0.1:34785)
0805 22:52:48.871975 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:48.871982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:48.872377 (+ 395us) server_negotiation.cc:407] Connection header received
0805 22:52:48.872554 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:48.872560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:48.872657 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:48.872787 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:48.873435 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:48.874580 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:48.876427 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:48.877137 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:48.877297 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:48.878668 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:48.878676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:48.878681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:48.878709 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:48.878765 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:48.878787 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:48.878918 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:48.879175 (+ 257us) server_negotiation.cc:299] Negotiation successful
0805 22:52:48.879247 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 22:52:48.880090 2878 leader_election.cc:302] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 729d6d6cd3a745d281dcfa78f6fd8dea; no voters:
I20260805 22:52:48.880828 2878 leader_election.cc:288] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:48.880995 2888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:48.882040 2888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [term 1 LEADER]: Becoming Leader. State: Replica: 729d6d6cd3a745d281dcfa78f6fd8dea, State: Running, Role: LEADER
I20260805 22:52:48.882627 2888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d6d6cd3a745d281dcfa78f6fd8dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34785 } }
I20260805 22:52:48.883162 2878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:48.885581 2889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "729d6d6cd3a745d281dcfa78f6fd8dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d6d6cd3a745d281dcfa78f6fd8dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34785 } } }
I20260805 22:52:48.886211 2889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:48.888134 2900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:48.886214 2893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729d6d6cd3a745d281dcfa78f6fd8dea. Latest consensus state: current_term: 1 leader_uuid: "729d6d6cd3a745d281dcfa78f6fd8dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729d6d6cd3a745d281dcfa78f6fd8dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34785 } } }
I20260805 22:52:48.889943 2893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:48.894372 2900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:48.904858 2900 catalog_manager.cc:1347] Generated new cluster ID: 9c57fc391f9941c28a320d56b3886386
I20260805 22:52:48.905021 2900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:48.920554 2900 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:48.922122 2900 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:48.929059 2900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 729d6d6cd3a745d281dcfa78f6fd8dea: Generated new TSK 0
I20260805 22:52:48.930039 2900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:48.939715 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2796
W20260805 22:52:48.941357 2871 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.224:56205
W20260805 22:52:48.942271 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:48.942451 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 2796
W20260805 22:52:48.965354 2774 connection.cc:535] client connection to 127.0.0.1:34785 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34785): Connection reset by peer (error 104)
2026-08-05T22:52:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:49 dist-test-slave-7lbf krb5kdc[2917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:49 dist-test-slave-7lbf krb5kdc[2917](info): set up 2 sockets
Aug 05 22:52:49 dist-test-slave-7lbf krb5kdc[2917](info): commencing operation
krb5kdc: starting...
W20260805 22:52:50.170744 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:50 dist-test-slave-7lbf krb5kdc[2917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 22:52:50.540892 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:50.917233 2934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:50.917600 2934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:50.917702 2934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:50.925340 2934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:50.925487 2934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:50.925603 2934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:50.925678 2934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:50.925725 2934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:50.936899 2934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:50.938478 2934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:50.939934 2934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:50.952984 2940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:50.952950 2942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:50.953097 2944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:50.953869 2934 server_base.cc:1034] running on GCE node
I20260805 22:52:50.954754 2934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:50.956382 2934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:50.957618 2934 hybrid_clock.cc:648] HybridClock initialized: now 1785970370957555 us; error 67 us; skew 500 ppm
I20260805 22:52:50.958005 2934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:50 dist-test-slave-7lbf krb5kdc[2917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:50.965652 2934 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 22:52:50.967509 2934 webserver.cc:460] Webserver started at http://127.0.0.1:36785/ using document root <none> and password file <none>
I20260805 22:52:50.968278 2934 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:50.968416 2934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:50.968722 2934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:50.972149 2934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "99ae0a9e72104a0d8dea1db6d83b29fd"
format_stamp: "Formatted at 2026-08-05 22:52:50 on dist-test-slave-7lbf"
server_key: "1065569b198a22813f099fc2c66bd519"
server_key_iv: "f94b05a444992a610d0f0a7519d65740"
server_key_version: "encryptionkey@0"
I20260805 22:52:50.973012 2934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "99ae0a9e72104a0d8dea1db6d83b29fd"
format_stamp: "Formatted at 2026-08-05 22:52:50 on dist-test-slave-7lbf"
server_key: "1065569b198a22813f099fc2c66bd519"
server_key_iv: "f94b05a444992a610d0f0a7519d65740"
server_key_version: "encryptionkey@0"
I20260805 22:52:50.979003 2934 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260805 22:52:50.983161 2952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:50.984774 2934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 22:52:50.984970 2934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "99ae0a9e72104a0d8dea1db6d83b29fd"
format_stamp: "Formatted at 2026-08-05 22:52:50 on dist-test-slave-7lbf"
server_key: "1065569b198a22813f099fc2c66bd519"
server_key_iv: "f94b05a444992a610d0f0a7519d65740"
server_key_version: "encryptionkey@0"
I20260805 22:52:50.985255 2934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:50.991889 2934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:50.996598 2934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:50.996922 2934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:51.016288 2934 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41943
I20260805 22:52:51.016275 3020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41943 every 8 connection(s)
I20260805 22:52:51.018091 2934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:51.022420 3022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:51.028522 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 2934
I20260805 22:52:51.028733 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:51.029224 4208 external_mini_cluster.cc:1428] Setting key 3a4f7cb133a008ab1523b5e8ec41ff33
I20260805 22:52:51.034417 3022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd: Bootstrap starting.
I20260805 22:52:51.038301 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:51.031546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:34671 (local address 127.0.0.1:41943)
0805 22:52:51.032541 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:51.032555 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:51.032602 (+ 47us) server_negotiation.cc:407] Connection header received
0805 22:52:51.034442 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:51.034471 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:51.034896 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:51.035259 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:51.036511 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:51.036519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:51.036535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:51.036553 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:51.036622 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:51.036645 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:51.036842 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:51.037000 (+ 158us) server_negotiation.cc:299] Negotiation successful
0805 22:52:51.037206 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":449,"threads_started":1}
I20260805 22:52:51.038352 3022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:51.039566 3022 log.cc:824] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:51.042593 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:51.043038 3022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd: No bootstrap required, opened a new log
I20260805 22:52:51.050711 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:51.048023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:49081 (local address 127.0.0.1:41943)
0805 22:52:51.048244 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:51.048253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:51.048533 (+ 280us) server_negotiation.cc:407] Connection header received
0805 22:52:51.048702 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:51.048708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:51.048781 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:51.048899 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:51.049897 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:51.049903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:51.049908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:51.049931 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:51.049993 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:51.050020 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:51.050160 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:51.050297 (+ 137us) server_negotiation.cc:299] Negotiation successful
0805 22:52:51.050376 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260805 22:52:51.053303 3022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ae0a9e72104a0d8dea1db6d83b29fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260805 22:52:51.053787 3022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:51.053875 3022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ae0a9e72104a0d8dea1db6d83b29fd, State: Initialized, Role: FOLLOWER
I20260805 22:52:51.054443 3022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ae0a9e72104a0d8dea1db6d83b29fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260805 22:52:51.054704 3022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:51.054813 3022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:51.054955 3022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:51.057310 3022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ae0a9e72104a0d8dea1db6d83b29fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260805 22:52:51.057964 3022 leader_election.cc:302] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99ae0a9e72104a0d8dea1db6d83b29fd; no voters:
I20260805 22:52:51.058583 3022 leader_election.cc:288] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:51.058722 3033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:51.059403 3033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [term 1 LEADER]: Becoming Leader. State: Replica: 99ae0a9e72104a0d8dea1db6d83b29fd, State: Running, Role: LEADER
I20260805 22:52:51.060014 3033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ae0a9e72104a0d8dea1db6d83b29fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } }
I20260805 22:52:51.060794 3022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:51.064576 3035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99ae0a9e72104a0d8dea1db6d83b29fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ae0a9e72104a0d8dea1db6d83b29fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } } }
I20260805 22:52:51.065153 3035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:51.064421 3036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99ae0a9e72104a0d8dea1db6d83b29fd. Latest consensus state: current_term: 1 leader_uuid: "99ae0a9e72104a0d8dea1db6d83b29fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ae0a9e72104a0d8dea1db6d83b29fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41943 } } }
I20260805 22:52:51.067621 3036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:51.068498 3046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:51.074635 3046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:51.084868 3046 catalog_manager.cc:1347] Generated new cluster ID: 3f6a2dce7e734f40baacb7923b5c3df8
I20260805 22:52:51.085016 3046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:51.096751 3046 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:51.097348 3046 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:51.108273 3046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99ae0a9e72104a0d8dea1db6d83b29fd: Generated new TSK 0
I20260805 22:52:51.109576 3046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:51.115798 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2934
W20260805 22:52:51.117076 3014 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.224:34671
W20260805 22:52:51.117965 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 2934: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:51.118115 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 2934
2026-08-05T22:52:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2183 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:51 dist-test-slave-7lbf krb5kdc[3059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:51 dist-test-slave-7lbf krb5kdc[3059](info): set up 2 sockets
Aug 05 22:52:51 dist-test-slave-7lbf krb5kdc[3059](info): commencing operation
krb5kdc: starting...
W20260805 22:52:52.367280 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.005s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:52 dist-test-slave-7lbf krb5kdc[3059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:52:52.737553 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:40833
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:40833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:53.113119 3074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:53.113474 3074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:53.113638 3074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:53.121306 3074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:53.121474 3074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:53.121577 3074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:53.121663 3074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:53.121727 3074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:53.132766 3074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:40833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:40833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:53.134392 3074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:53.135848 3074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:53.148630 3080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:53.148519 3081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:53.148567 3085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:53.149902 3074 server_base.cc:1034] running on GCE node
I20260805 22:52:53.150852 3074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:53.152508 3074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:53.153738 3074 hybrid_clock.cc:648] HybridClock initialized: now 1785970373153691 us; error 45 us; skew 500 ppm
I20260805 22:52:53.154146 3074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:53 dist-test-slave-7lbf krb5kdc[3059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970373, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:53.157581 3074 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:52:53.159368 3074 webserver.cc:460] Webserver started at http://127.4.28.62:35295/ using document root <none> and password file <none>
I20260805 22:52:53.160148 3074 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:53.160284 3074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:53.160605 3074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:53.163954 3074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "230051d23d394a6eb48a4ffd9323f99c"
format_stamp: "Formatted at 2026-08-05 22:52:53 on dist-test-slave-7lbf"
server_key: "929088f9aeb7533ef14a02da6ebe3fbf"
server_key_iv: "0f738e2c209abf2f4e87a8f1b896ca7d"
server_key_version: "encryptionkey@0"
I20260805 22:52:53.164849 3074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "230051d23d394a6eb48a4ffd9323f99c"
format_stamp: "Formatted at 2026-08-05 22:52:53 on dist-test-slave-7lbf"
server_key: "929088f9aeb7533ef14a02da6ebe3fbf"
server_key_iv: "0f738e2c209abf2f4e87a8f1b896ca7d"
server_key_version: "encryptionkey@0"
I20260805 22:52:53.171074 3074 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.005s
I20260805 22:52:53.175323 3096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:53.176846 3074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260805 22:52:53.177062 3074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "230051d23d394a6eb48a4ffd9323f99c"
format_stamp: "Formatted at 2026-08-05 22:52:53 on dist-test-slave-7lbf"
server_key: "929088f9aeb7533ef14a02da6ebe3fbf"
server_key_iv: "0f738e2c209abf2f4e87a8f1b896ca7d"
server_key_version: "encryptionkey@0"
I20260805 22:52:53.177287 3074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:53.214182 3074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:53.219434 3074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:53.219785 3074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:53.240476 3074 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:40833
I20260805 22:52:53.240507 3165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:40833 every 8 connection(s)
I20260805 22:52:53.242137 3074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:53.245836 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 3074
I20260805 22:52:53.246016 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:53.246359 3169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:53.246521 4208 external_mini_cluster.cc:1428] Setting key b8baa2d3849d7914db6028f044941595
I20260805 22:52:53.262894 3169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c: Bootstrap starting.
Aug 05 22:52:53 dist-test-slave-7lbf krb5kdc[3059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:52:53.266764 3169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:53.268054 3169 log.cc:824] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:53.271284 3169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c: No bootstrap required, opened a new log
I20260805 22:52:53.273082 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:53.248557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55711 (local address 127.4.28.62:40833)
0805 22:52:53.249503 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:53.249553 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:53.249635 (+ 82us) server_negotiation.cc:407] Connection header received
0805 22:52:53.251626 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:53.251658 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:53.252195 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:53.252634 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:53.254848 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:53.256302 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:53.258337 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:53.259264 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:53.259443 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:53.264891 (+ 5448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:53.264903 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:53.264921 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:53.264987 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:53.268381 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:53.269336 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:53.269345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:53.269353 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:53.269588 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:53.270106 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:53.270111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:53.270113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:53.270716 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:53.271044 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:53.271596 (+ 552us) server_negotiation.cc:299] Negotiation successful
0805 22:52:53.271974 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":403,"threads_started":1}
I20260805 22:52:53.277284 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:53.279706 3169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230051d23d394a6eb48a4ffd9323f99c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40833 } }
I20260805 22:52:53.280128 3169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:53.280238 3169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230051d23d394a6eb48a4ffd9323f99c, State: Initialized, Role: FOLLOWER
I20260805 22:52:53.280947 3169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230051d23d394a6eb48a4ffd9323f99c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40833 } }
I20260805 22:52:53.281214 3169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:53.281369 3169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:53.281594 3169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:53.284847 3169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230051d23d394a6eb48a4ffd9323f99c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40833 } }
I20260805 22:52:53.285629 3169 leader_election.cc:302] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230051d23d394a6eb48a4ffd9323f99c; no voters:
I20260805 22:52:53.286388 3169 leader_election.cc:288] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:53.286541 3183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:53.287201 3183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [term 1 LEADER]: Becoming Leader. State: Replica: 230051d23d394a6eb48a4ffd9323f99c, State: Running, Role: LEADER
I20260805 22:52:53.287914 3183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230051d23d394a6eb48a4ffd9323f99c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40833 } }
I20260805 22:52:53.291299 3185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "230051d23d394a6eb48a4ffd9323f99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230051d23d394a6eb48a4ffd9323f99c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40833 } } }
I20260805 22:52:53.291823 3185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:53.293406 3184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230051d23d394a6eb48a4ffd9323f99c. Latest consensus state: current_term: 1 leader_uuid: "230051d23d394a6eb48a4ffd9323f99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230051d23d394a6eb48a4ffd9323f99c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 40833 } } }
I20260805 22:52:53.293757 3184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:53.295941 3169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:53.302134 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:53.281849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44995 (local address 127.4.28.62:40833)
0805 22:52:53.282040 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:53.282046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:53.282658 (+ 612us) server_negotiation.cc:407] Connection header received
0805 22:52:53.282833 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:53.282839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:53.282931 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:53.283057 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:53.284991 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:53.286123 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:53.288012 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:53.288783 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:53.288946 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:53.292645 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:53.292655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:53.292661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:52:53.292721 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:53.296723 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:53.298316 (+ 1593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:53.298324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:53.298329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:53.298595 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:52:53.299295 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:52:53.299301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:52:53.299305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:52:53.299766 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:53.300079 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:53.301447 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0805 22:52:53.301711 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 22:52:53.304381 3188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:53.310469 3188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:53.321666 3188 catalog_manager.cc:1347] Generated new cluster ID: 60880d2a3a7d4b08b0518cafa90a7b12
I20260805 22:52:53.321802 3188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:53.334477 3188 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:53.335944 3188 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:53.342907 3188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230051d23d394a6eb48a4ffd9323f99c: Generated new TSK 0
I20260805 22:52:53.343941 3188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:53.354791 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3074
I20260805 22:52:53.356163 3160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:52:53.449191 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 3074
W20260805 22:52:53.475466 3056 connection.cc:535] client connection to 127.4.28.62:40833 recv error: Network error: failed to read from TLS socket (remote: 127.4.28.62:40833): Connection reset by peer (error 104)
2026-08-05T22:52:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:53 dist-test-slave-7lbf krb5kdc[3216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:53 dist-test-slave-7lbf krb5kdc[3216](info): set up 2 sockets
Aug 05 22:52:53 dist-test-slave-7lbf krb5kdc[3216](info): commencing operation
krb5kdc: starting...
W20260805 22:52:54.699569 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.002s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:54 dist-test-slave-7lbf krb5kdc[3216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:52:55.079423 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44367
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:44367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:55.451625 3231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:55.451953 3231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:55.452073 3231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:55.459679 3231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:55.459815 3231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:55.459861 3231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:55.459932 3231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:55.459997 3231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:55.470669 3231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:44367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:44367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:55.472122 3231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:55.473593 3231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:55.485872 3237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:55.485930 3240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:55.486171 3236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:55.486686 3231 server_base.cc:1034] running on GCE node
I20260805 22:52:55.487543 3231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:55.489123 3231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:55.490366 3231 hybrid_clock.cc:648] HybridClock initialized: now 1785970375490315 us; error 42 us; skew 500 ppm
I20260805 22:52:55.490763 3231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:55 dist-test-slave-7lbf krb5kdc[3216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970375, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:55.494033 3231 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:52:55.495821 3231 webserver.cc:460] Webserver started at http://127.4.28.62:35931/ using document root <none> and password file <none>
I20260805 22:52:55.496595 3231 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:55.496727 3231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:55.497058 3231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:55.500460 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "f032a7a5eba1430b821b03a116c849e8"
format_stamp: "Formatted at 2026-08-05 22:52:55 on dist-test-slave-7lbf"
server_key: "eb25944d45f3766b046ba73bd434df17"
server_key_iv: "5f9a4c4b4bee4006924a1d653819c1e6"
server_key_version: "encryptionkey@0"
I20260805 22:52:55.501358 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "f032a7a5eba1430b821b03a116c849e8"
format_stamp: "Formatted at 2026-08-05 22:52:55 on dist-test-slave-7lbf"
server_key: "eb25944d45f3766b046ba73bd434df17"
server_key_iv: "5f9a4c4b4bee4006924a1d653819c1e6"
server_key_version: "encryptionkey@0"
I20260805 22:52:55.507664 3231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260805 22:52:55.511898 3250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:55.513793 3231 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:52:55.514019 3231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "f032a7a5eba1430b821b03a116c849e8"
format_stamp: "Formatted at 2026-08-05 22:52:55 on dist-test-slave-7lbf"
server_key: "eb25944d45f3766b046ba73bd434df17"
server_key_iv: "5f9a4c4b4bee4006924a1d653819c1e6"
server_key_version: "encryptionkey@0"
I20260805 22:52:55.514237 3231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:55.528384 3231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:55.532912 3231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:55.533206 3231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:55.552296 3231 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:44367
I20260805 22:52:55.552318 3319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:44367 every 8 connection(s)
I20260805 22:52:55.553825 3231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:55.557585 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 3231
I20260805 22:52:55.557785 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:55.558192 3321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:55.558321 4208 external_mini_cluster.cc:1428] Setting key c10fbe676fd95c412e418d11fe1ef53d
I20260805 22:52:55.573624 3321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8: Bootstrap starting.
I20260805 22:52:55.574039 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:55.560430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57133 (local address 127.4.28.62:44367)
0805 22:52:55.561315 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:55.561331 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:55.561388 (+ 57us) server_negotiation.cc:407] Connection header received
0805 22:52:55.563481 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:55.563511 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:55.564004 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:55.564401 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:55.565173 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:55.566722 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:55.568768 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:55.569667 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:55.569881 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:55.571363 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:55.571372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:55.571390 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:55.572024 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:55.572117 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:55.572152 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:55.572282 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:55.572601 (+ 319us) server_negotiation.cc:299] Negotiation successful
0805 22:52:55.572817 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":407,"threads_started":1}
I20260805 22:52:55.577551 3321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:55.578449 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:55.578810 3321 log.cc:824] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:55.582090 3321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8: No bootstrap required, opened a new log
I20260805 22:52:55.590919 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:55.583369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59743 (local address 127.4.28.62:44367)
0805 22:52:55.583598 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:55.583604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:55.584037 (+ 433us) server_negotiation.cc:407] Connection header received
0805 22:52:55.584210 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:55.584216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:55.584307 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:55.584431 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:55.584933 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:55.586175 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:55.587616 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:52:55.588367 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:52:55.588586 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:52:55.590066 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:55.590071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:55.590074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:55.590103 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:55.590172 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:55.590190 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:55.590299 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:55.590552 (+ 253us) server_negotiation.cc:299] Negotiation successful
0805 22:52:55.590621 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260805 22:52:55.592334 3321 raft_consensus.cc:348] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f032a7a5eba1430b821b03a116c849e8" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44367 } }
I20260805 22:52:55.592905 3321 raft_consensus.cc:374] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:55.593014 3321 raft_consensus.cc:729] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f032a7a5eba1430b821b03a116c849e8, State: Initialized, Role: FOLLOWER
I20260805 22:52:55.593711 3321 consensus_queue.cc:260] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f032a7a5eba1430b821b03a116c849e8" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44367 } }
I20260805 22:52:55.594015 3321 raft_consensus.cc:388] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:55.594193 3321 raft_consensus.cc:482] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:55.594414 3321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:55.597002 3321 raft_consensus.cc:504] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f032a7a5eba1430b821b03a116c849e8" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44367 } }
I20260805 22:52:55.597692 3321 leader_election.cc:302] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f032a7a5eba1430b821b03a116c849e8; no voters:
I20260805 22:52:55.598430 3321 leader_election.cc:288] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:55.598589 3335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:55.599849 3335 raft_consensus.cc:686] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [term 1 LEADER]: Becoming Leader. State: Replica: f032a7a5eba1430b821b03a116c849e8, State: Running, Role: LEADER
I20260805 22:52:55.600543 3335 consensus_queue.cc:237] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f032a7a5eba1430b821b03a116c849e8" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44367 } }
I20260805 22:52:55.600835 3321 sys_catalog.cc:564] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:55.603849 3337 sys_catalog.cc:455] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f032a7a5eba1430b821b03a116c849e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f032a7a5eba1430b821b03a116c849e8" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44367 } } }
I20260805 22:52:55.604336 3337 sys_catalog.cc:458] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:55.604084 3338 sys_catalog.cc:455] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f032a7a5eba1430b821b03a116c849e8. Latest consensus state: current_term: 1 leader_uuid: "f032a7a5eba1430b821b03a116c849e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f032a7a5eba1430b821b03a116c849e8" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 44367 } } }
I20260805 22:52:55.604466 3338 sys_catalog.cc:458] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:55.606604 3348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:55.611898 3348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:55.623644 3348 catalog_manager.cc:1347] Generated new cluster ID: f64c7a1cbcbe4dcfa04f6d5c9b5de712
I20260805 22:52:55.623796 3348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:55.648470 3348 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:55.650035 3348 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:55.659978 3348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f032a7a5eba1430b821b03a116c849e8: Generated new TSK 0
I20260805 22:52:55.660965 3348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:55.674587 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3231
W20260805 22:52:55.676214 3313 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57133
W20260805 22:52:55.677199 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3231: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:55.677381 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 3231
W20260805 22:52:55.706696 3212 connection.cc:535] client connection to 127.4.28.62:44367 recv error: Network error: failed to read from TLS socket (remote: 127.4.28.62:44367): Connection reset by peer (error 104)
2026-08-05T22:52:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:55 dist-test-slave-7lbf krb5kdc[3370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:55 dist-test-slave-7lbf krb5kdc[3370](info): set up 2 sockets
Aug 05 22:52:55 dist-test-slave-7lbf krb5kdc[3370](info): commencing operation
krb5kdc: starting...
W20260805 22:52:56.933904 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.007s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:57 dist-test-slave-7lbf krb5kdc[3370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:52:57.313267 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:41657
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:41657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:57.704156 3383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:57.704514 3383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:57.704653 3383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:57.712600 3383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:57.712741 3383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:57.712801 3383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:57.712893 3383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:57.712946 3383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:57.723928 3383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:41657
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:41657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:57.725425 3383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:57.726979 3383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:57.741119 3394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:57.741146 3398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:57.741268 3395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:57.741999 3383 server_base.cc:1034] running on GCE node
I20260805 22:52:57.742918 3383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:57.744524 3383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:57.745781 3383 hybrid_clock.cc:648] HybridClock initialized: now 1785970377745714 us; error 56 us; skew 500 ppm
I20260805 22:52:57.746249 3383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:57 dist-test-slave-7lbf krb5kdc[3370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970377, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:57.754623 3383 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:52:57.756572 3383 webserver.cc:460] Webserver started at http://127.4.28.62:36067/ using document root <none> and password file <none>
I20260805 22:52:57.757386 3383 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:57.757560 3383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:57.757896 3383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:57.761395 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "62b8b0833d7f4c178516ec8bd0b17308"
format_stamp: "Formatted at 2026-08-05 22:52:57 on dist-test-slave-7lbf"
server_key: "350253f0144518b8b17e880a1514c572"
server_key_iv: "fd1e76059882594de36de832cdacbceb"
server_key_version: "encryptionkey@0"
I20260805 22:52:57.762342 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "62b8b0833d7f4c178516ec8bd0b17308"
format_stamp: "Formatted at 2026-08-05 22:52:57 on dist-test-slave-7lbf"
server_key: "350253f0144518b8b17e880a1514c572"
server_key_iv: "fd1e76059882594de36de832cdacbceb"
server_key_version: "encryptionkey@0"
I20260805 22:52:57.768442 3383 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260805 22:52:57.772927 3407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:57.774658 3383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:52:57.774919 3383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "62b8b0833d7f4c178516ec8bd0b17308"
format_stamp: "Formatted at 2026-08-05 22:52:57 on dist-test-slave-7lbf"
server_key: "350253f0144518b8b17e880a1514c572"
server_key_iv: "fd1e76059882594de36de832cdacbceb"
server_key_version: "encryptionkey@0"
I20260805 22:52:57.775148 3383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:52:57.781895 3383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:52:57.786839 3383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:52:57.787161 3383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:52:57.807173 3383 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:41657
I20260805 22:52:57.807216 3472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:41657 every 8 connection(s)
I20260805 22:52:57.808725 3383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:52:57.812943 3473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:52:57.813900 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 3383
I20260805 22:52:57.814124 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:52:57.814705 4208 external_mini_cluster.cc:1428] Setting key 1f2879da3e6f32929b54a2203f3eef58
I20260805 22:52:57.823661 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:57.816825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39653 (local address 127.4.28.62:41657)
0805 22:52:57.817738 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:57.817751 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:57.817805 (+ 54us) server_negotiation.cc:407] Connection header received
0805 22:52:57.819772 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:57.819802 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:57.820241 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:57.820642 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:57.821883 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:57.821890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:57.821906 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:57.821926 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:57.822006 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:57.822035 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:57.822243 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:57.822362 (+ 119us) server_negotiation.cc:299] Negotiation successful
0805 22:52:57.822606 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":398,"threads_started":1}
I20260805 22:52:57.827512 3473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308: Bootstrap starting.
I20260805 22:52:57.828074 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:52:57.831825 3473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308: Neither blocks nor log segments found. Creating new log.
I20260805 22:52:57.833389 3473 log.cc:824] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308: Log is configured to *not* fsync() on all Append() calls
I20260805 22:52:57.836472 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:52:57.833789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52639 (local address 127.4.28.62:41657)
0805 22:52:57.833994 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0805 22:52:57.834002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:52:57.834435 (+ 433us) server_negotiation.cc:407] Connection header received
0805 22:52:57.834652 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:52:57.834657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:52:57.834721 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:52:57.834815 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:52:57.835793 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:52:57.835798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:52:57.835801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:52:57.835818 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:52:57.835861 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:52:57.835877 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:52:57.835992 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0805 22:52:57.836122 (+ 130us) server_negotiation.cc:299] Negotiation successful
0805 22:52:57.836185 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260805 22:52:57.836851 3473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308: No bootstrap required, opened a new log
I20260805 22:52:57.845608 3473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b8b0833d7f4c178516ec8bd0b17308" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41657 } }
I20260805 22:52:57.846117 3473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:52:57.846230 3473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b8b0833d7f4c178516ec8bd0b17308, State: Initialized, Role: FOLLOWER
I20260805 22:52:57.846903 3473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b8b0833d7f4c178516ec8bd0b17308" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41657 } }
I20260805 22:52:57.847169 3473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:52:57.847314 3473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:52:57.847487 3473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:52:57.850054 3473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b8b0833d7f4c178516ec8bd0b17308" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41657 } }
I20260805 22:52:57.850675 3473 leader_election.cc:302] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62b8b0833d7f4c178516ec8bd0b17308; no voters:
I20260805 22:52:57.851333 3473 leader_election.cc:288] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:52:57.851498 3483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:52:57.852206 3483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [term 1 LEADER]: Becoming Leader. State: Replica: 62b8b0833d7f4c178516ec8bd0b17308, State: Running, Role: LEADER
I20260805 22:52:57.852802 3483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b8b0833d7f4c178516ec8bd0b17308" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41657 } }
I20260805 22:52:57.853785 3473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:52:57.856453 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62b8b0833d7f4c178516ec8bd0b17308. Latest consensus state: current_term: 1 leader_uuid: "62b8b0833d7f4c178516ec8bd0b17308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b8b0833d7f4c178516ec8bd0b17308" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41657 } } }
I20260805 22:52:57.856441 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62b8b0833d7f4c178516ec8bd0b17308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b8b0833d7f4c178516ec8bd0b17308" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 41657 } } }
I20260805 22:52:57.856938 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:57.856953 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308 [sys.catalog]: This master's current role is: LEADER
I20260805 22:52:57.859165 3493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:52:57.864442 3493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:52:57.874307 3493 catalog_manager.cc:1347] Generated new cluster ID: ce2ef5e7a6994f01b41131c9e239279a
I20260805 22:52:57.874462 3493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:52:57.900161 3493 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:52:57.900763 3493 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:52:57.911473 3493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62b8b0833d7f4c178516ec8bd0b17308: Generated new TSK 0
I20260805 22:52:57.912698 3493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:52:57.933071 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3383
W20260805 22:52:57.934533 3467 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39653
W20260805 22:52:57.935444 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3383: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:52:57.935595 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 3383
2026-08-05T22:52:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:52:58 dist-test-slave-7lbf krb5kdc[3509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:52:58 dist-test-slave-7lbf krb5kdc[3509](info): set up 2 sockets
Aug 05 22:52:58 dist-test-slave-7lbf krb5kdc[3509](info): commencing operation
krb5kdc: starting...
W20260805 22:52:59.195111 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.002s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:52:59 dist-test-slave-7lbf krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:52:59.565073 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:35187
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:52:59.935822 3527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:52:59.936169 3527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:52:59.936268 3527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:52:59.943979 3527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:52:59.944118 3527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:52:59.944175 3527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:52:59.944237 3527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:52:59.944283 3527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:52:59.954823 3527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:35187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:52:59.956243 3527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:52:59.957676 3527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:52:59.970350 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:59.970311 3537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:52:59.970386 3535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:52:59.971282 3527 server_base.cc:1034] running on GCE node
I20260805 22:52:59.972153 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:52:59.973745 3527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:52:59.974984 3527 hybrid_clock.cc:648] HybridClock initialized: now 1785970379974926 us; error 53 us; skew 500 ppm
I20260805 22:52:59.975368 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:52:59 dist-test-slave-7lbf krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970379, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:52:59.978724 3527 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:52:59.980386 3527 webserver.cc:460] Webserver started at http://127.4.28.62:43415/ using document root <none> and password file <none>
I20260805 22:52:59.981184 3527 fs_manager.cc:359] Metadata directory not provided
I20260805 22:52:59.981328 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:52:59.981703 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:52:59.985028 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "30a30a3aae714aec9c7720efb841d19c"
format_stamp: "Formatted at 2026-08-05 22:52:59 on dist-test-slave-7lbf"
server_key: "6c43da65627695d1a936d618394922db"
server_key_iv: "6a6e3f7870a99e5dff6eac88eb15af8d"
server_key_version: "encryptionkey@0"
I20260805 22:52:59.985950 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "30a30a3aae714aec9c7720efb841d19c"
format_stamp: "Formatted at 2026-08-05 22:52:59 on dist-test-slave-7lbf"
server_key: "6c43da65627695d1a936d618394922db"
server_key_iv: "6a6e3f7870a99e5dff6eac88eb15af8d"
server_key_version: "encryptionkey@0"
I20260805 22:52:59.991986 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260805 22:52:59.996182 3546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:52:59.997987 3527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:52:59.998230 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "30a30a3aae714aec9c7720efb841d19c"
format_stamp: "Formatted at 2026-08-05 22:52:59 on dist-test-slave-7lbf"
server_key: "6c43da65627695d1a936d618394922db"
server_key_iv: "6a6e3f7870a99e5dff6eac88eb15af8d"
server_key_version: "encryptionkey@0"
I20260805 22:52:59.998468 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:53:00.023274 3527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:53:00.028136 3527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:53:00.028474 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:53:00.047319 3527 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:35187
I20260805 22:53:00.047350 3613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:35187 every 8 connection(s)
I20260805 22:53:00.048828 3527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:53:00.052625 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 3527
I20260805 22:53:00.052821 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:53:00.053217 3616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:53:00.053295 4208 external_mini_cluster.cc:1428] Setting key 4669f04f485cbffb831cfc32136308f1
I20260805 22:53:00.069314 3616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c: Bootstrap starting.
Aug 05 22:53:00 dist-test-slave-7lbf krb5kdc[3509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:53:00.073623 3616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c: Neither blocks nor log segments found. Creating new log.
I20260805 22:53:00.075232 3616 log.cc:824] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c: Log is configured to *not* fsync() on all Append() calls
I20260805 22:53:00.078562 3616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c: No bootstrap required, opened a new log
I20260805 22:53:00.080330 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:00.055378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53619 (local address 127.4.28.62:35187)
0805 22:53:00.056345 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:00.056366 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:00.056420 (+ 54us) server_negotiation.cc:407] Connection header received
0805 22:53:00.058815 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:00.058854 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:00.059534 (+ 680us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:00.060091 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:00.061655 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:00.063703 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:00.065423 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:00.066687 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:00.066922 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:00.071437 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:00.071462 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:00.071486 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:00.071555 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:00.075620 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:00.076605 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:00.076615 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:00.076623 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:00.076833 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:00.077286 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:00.077290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:00.077292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:00.077943 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:00.078281 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:00.078783 (+ 502us) server_negotiation.cc:299] Negotiation successful
0805 22:53:00.079256 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":390,"threads_started":1}
I20260805 22:53:00.085093 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:53:00.087630 3616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a30a3aae714aec9c7720efb841d19c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35187 } }
I20260805 22:53:00.088168 3616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:53:00.088294 3616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a30a3aae714aec9c7720efb841d19c, State: Initialized, Role: FOLLOWER
I20260805 22:53:00.088961 3616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a30a3aae714aec9c7720efb841d19c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35187 } }
I20260805 22:53:00.089222 3616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:53:00.089375 3616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:53:00.089604 3616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:53:00.092288 3616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a30a3aae714aec9c7720efb841d19c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35187 } }
I20260805 22:53:00.092886 3616 leader_election.cc:302] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30a30a3aae714aec9c7720efb841d19c; no voters:
I20260805 22:53:00.093628 3616 leader_election.cc:288] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:53:00.093753 3628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:53:00.095306 3628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [term 1 LEADER]: Becoming Leader. State: Replica: 30a30a3aae714aec9c7720efb841d19c, State: Running, Role: LEADER
I20260805 22:53:00.096060 3628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a30a3aae714aec9c7720efb841d19c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35187 } }
I20260805 22:53:00.096352 3616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:53:00.103227 3629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30a30a3aae714aec9c7720efb841d19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a30a3aae714aec9c7720efb841d19c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35187 } } }
I20260805 22:53:00.103834 3629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:00.104614 3630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a30a3aae714aec9c7720efb841d19c. Latest consensus state: current_term: 1 leader_uuid: "30a30a3aae714aec9c7720efb841d19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a30a3aae714aec9c7720efb841d19c" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35187 } } }
I20260805 22:53:00.105551 3630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:00.106570 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:00.089843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54537 (local address 127.4.28.62:35187)
0805 22:53:00.090051 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:00.090058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:00.090465 (+ 407us) server_negotiation.cc:407] Connection header received
0805 22:53:00.090708 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:00.090718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:00.090798 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:00.090896 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:00.092402 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:00.093214 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:00.095069 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:00.095761 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:00.095902 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:00.098679 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:00.098687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:00.098692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:00.098739 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:00.101764 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:00.102870 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:00.102876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:00.102880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:00.103079 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:00.103602 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:00.103608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:00.103611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:00.104117 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:00.104414 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:00.105873 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0805 22:53:00.106167 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 22:53:00.111564 3635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:53:00.117156 3635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:53:00.127445 3635 catalog_manager.cc:1347] Generated new cluster ID: e179d41bb9ca4086832a01bd7779fd80
I20260805 22:53:00.127588 3635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:53:00.163659 3635 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:53:00.165542 3635 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:53:00.176272 3635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30a30a3aae714aec9c7720efb841d19c: Generated new TSK 0
I20260805 22:53:00.177659 3635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:53:00.194250 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3527
I20260805 22:53:00.195628 3603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:53:00.294559 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 3527
2026-08-05T22:53:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:53:00 dist-test-slave-7lbf krb5kdc[3654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:53:00 dist-test-slave-7lbf krb5kdc[3654](info): set up 2 sockets
Aug 05 22:53:00 dist-test-slave-7lbf krb5kdc[3654](info): commencing operation
krb5kdc: starting...
W20260805 22:53:01.540159 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.003s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:53:01 dist-test-slave-7lbf krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:53:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:53:01.921468 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:39143
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:39143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:53:02.299793 3669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:53:02.300148 3669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:53:02.300246 3669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:53:02.307780 3669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:53:02.307915 3669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:53:02.307988 3669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:53:02.308050 3669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:53:02.308094 3669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:53:02.318523 3669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:39143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:39143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:53:02.319936 3669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:53:02.321365 3669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:53:02.334616 3674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:02.334697 3677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:02.334726 3675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:53:02.335598 3669 server_base.cc:1034] running on GCE node
I20260805 22:53:02.336532 3669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:53:02.338304 3669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:53:02.339583 3669 hybrid_clock.cc:648] HybridClock initialized: now 1785970382339517 us; error 76 us; skew 500 ppm
I20260805 22:53:02.340020 3669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:53:02 dist-test-slave-7lbf krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970382, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:53:02.343672 3669 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:53:02.345367 3669 webserver.cc:460] Webserver started at http://127.4.28.62:37931/ using document root <none> and password file <none>
I20260805 22:53:02.346189 3669 fs_manager.cc:359] Metadata directory not provided
I20260805 22:53:02.346309 3669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:53:02.346596 3669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:53:02.350132 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "7aefc7e0be604d919ca3c1d61e0abc04"
format_stamp: "Formatted at 2026-08-05 22:53:02 on dist-test-slave-7lbf"
server_key: "9eb5c22b2afe349d91a2369c81af36ae"
server_key_iv: "517548d08db406be72edcabc3e86b2a5"
server_key_version: "encryptionkey@0"
I20260805 22:53:02.351004 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "7aefc7e0be604d919ca3c1d61e0abc04"
format_stamp: "Formatted at 2026-08-05 22:53:02 on dist-test-slave-7lbf"
server_key: "9eb5c22b2afe349d91a2369c81af36ae"
server_key_iv: "517548d08db406be72edcabc3e86b2a5"
server_key_version: "encryptionkey@0"
I20260805 22:53:02.357133 3669 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260805 22:53:02.361251 3686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:53:02.362897 3669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:53:02.363142 3669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "7aefc7e0be604d919ca3c1d61e0abc04"
format_stamp: "Formatted at 2026-08-05 22:53:02 on dist-test-slave-7lbf"
server_key: "9eb5c22b2afe349d91a2369c81af36ae"
server_key_iv: "517548d08db406be72edcabc3e86b2a5"
server_key_version: "encryptionkey@0"
I20260805 22:53:02.363360 3669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:53:02.379658 3669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:53:02.384629 3669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:53:02.384940 3669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:53:02.405117 3669 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:39143
I20260805 22:53:02.405148 3752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:39143 every 8 connection(s)
I20260805 22:53:02.406714 3669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:53:02.409938 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 3669
I20260805 22:53:02.410161 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:53:02.410624 4208 external_mini_cluster.cc:1428] Setting key b49fe80100d41eb7bb881cb6ab851c84
I20260805 22:53:02.411024 3753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:53:02.427670 3753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04: Bootstrap starting.
I20260805 22:53:02.427738 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:02.412637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59087 (local address 127.4.28.62:39143)
0805 22:53:02.413583 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:02.413601 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:02.413663 (+ 62us) server_negotiation.cc:407] Connection header received
0805 22:53:02.416256 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:02.416299 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:02.417014 (+ 715us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:02.417608 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:02.418251 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:02.420349 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:02.422005 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:02.423111 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:02.423329 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:02.424540 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:02.424552 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:02.424584 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:53:02.425423 (+ 839us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:02.425582 (+ 159us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:53:02.425619 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:02.425750 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:02.425957 (+ 207us) server_negotiation.cc:299] Negotiation successful
0805 22:53:02.426276 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":371,"threads_started":1}
I20260805 22:53:02.431420 3753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04: Neither blocks nor log segments found. Creating new log.
I20260805 22:53:02.432185 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:53:02.432789 3753 log.cc:824] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04: Log is configured to *not* fsync() on all Append() calls
I20260805 22:53:02.436707 3753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04: No bootstrap required, opened a new log
I20260805 22:53:02.444633 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:02.437508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44815 (local address 127.4.28.62:39143)
0805 22:53:02.437731 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:02.437737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:02.438154 (+ 417us) server_negotiation.cc:407] Connection header received
0805 22:53:02.438339 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:02.438344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:02.438430 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:02.438559 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:02.439076 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:02.440239 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:02.441718 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:02.442459 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:02.442599 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:02.443796 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:02.443803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:02.443808 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:53:02.443843 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:02.443896 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:53:02.443917 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:02.444008 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:02.444148 (+ 140us) server_negotiation.cc:299] Negotiation successful
0805 22:53:02.444257 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 22:53:02.446749 3753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aefc7e0be604d919ca3c1d61e0abc04" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39143 } }
I20260805 22:53:02.447151 3753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:53:02.447230 3753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aefc7e0be604d919ca3c1d61e0abc04, State: Initialized, Role: FOLLOWER
I20260805 22:53:02.447782 3753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aefc7e0be604d919ca3c1d61e0abc04" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39143 } }
I20260805 22:53:02.448007 3753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:53:02.448113 3753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:53:02.448246 3753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:53:02.450771 3753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aefc7e0be604d919ca3c1d61e0abc04" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39143 } }
I20260805 22:53:02.451292 3753 leader_election.cc:302] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aefc7e0be604d919ca3c1d61e0abc04; no voters:
I20260805 22:53:02.451890 3753 leader_election.cc:288] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:53:02.452047 3766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:53:02.453089 3766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [term 1 LEADER]: Becoming Leader. State: Replica: 7aefc7e0be604d919ca3c1d61e0abc04, State: Running, Role: LEADER
I20260805 22:53:02.453725 3766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aefc7e0be604d919ca3c1d61e0abc04" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39143 } }
I20260805 22:53:02.454246 3753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:53:02.456573 3767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aefc7e0be604d919ca3c1d61e0abc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aefc7e0be604d919ca3c1d61e0abc04" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39143 } } }
I20260805 22:53:02.457044 3767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:02.456887 3769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aefc7e0be604d919ca3c1d61e0abc04. Latest consensus state: current_term: 1 leader_uuid: "7aefc7e0be604d919ca3c1d61e0abc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aefc7e0be604d919ca3c1d61e0abc04" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 39143 } } }
I20260805 22:53:02.457213 3769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04 [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:02.459249 3779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:53:02.463925 3779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:53:02.473950 3779 catalog_manager.cc:1347] Generated new cluster ID: 57476eb61c6c41d597fa934a588c3aaa
I20260805 22:53:02.474102 3779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:53:02.506582 3779 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:53:02.508158 3779 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:53:02.520709 3779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aefc7e0be604d919ca3c1d61e0abc04: Generated new TSK 0
I20260805 22:53:02.521795 3779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:53:02.540088 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3669
W20260805 22:53:02.541858 3748 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59087
W20260805 22:53:02.542730 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3669: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:53:02.542908 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 3669
2026-08-05T22:53:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:53:02 dist-test-slave-7lbf krb5kdc[3795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:53:02 dist-test-slave-7lbf krb5kdc[3795](info): set up 2 sockets
Aug 05 22:53:02 dist-test-slave-7lbf krb5kdc[3795](info): commencing operation
krb5kdc: starting...
W20260805 22:53:03.793100 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:53:04 dist-test-slave-7lbf krb5kdc[3795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:53:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:53:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:53:04.173404 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:35875
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:35875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:53:04.544588 3812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:53:04.544945 3812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:53:04.545070 3812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:53:04.552693 3812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:53:04.552835 3812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:53:04.552913 3812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:53:04.553004 3812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:53:04.553059 3812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:53:04.563791 3812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:35875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:35875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:53:04.565320 3812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:53:04.566918 3812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:53:04.580560 3818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:04.580492 3821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:04.580520 3819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:53:04.581457 3812 server_base.cc:1034] running on GCE node
I20260805 22:53:04.582410 3812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:53:04.584151 3812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:53:04.585409 3812 hybrid_clock.cc:648] HybridClock initialized: now 1785970384585349 us; error 58 us; skew 500 ppm
I20260805 22:53:04.585858 3812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:53:04 dist-test-slave-7lbf krb5kdc[3795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970384, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:53:04.594288 3812 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:53:04.596127 3812 webserver.cc:460] Webserver started at http://127.4.28.62:43935/ using document root <none> and password file <none>
I20260805 22:53:04.596949 3812 fs_manager.cc:359] Metadata directory not provided
I20260805 22:53:04.597082 3812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:53:04.597429 3812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:53:04.600941 3812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d"
format_stamp: "Formatted at 2026-08-05 22:53:04 on dist-test-slave-7lbf"
server_key: "dcd45835708f004ea417265e228f4515"
server_key_iv: "71240c6691b6a7e615435cd035010508"
server_key_version: "encryptionkey@0"
I20260805 22:53:04.601917 3812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d"
format_stamp: "Formatted at 2026-08-05 22:53:04 on dist-test-slave-7lbf"
server_key: "dcd45835708f004ea417265e228f4515"
server_key_iv: "71240c6691b6a7e615435cd035010508"
server_key_version: "encryptionkey@0"
I20260805 22:53:04.608271 3812 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260805 22:53:04.612639 3829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:53:04.614349 3812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:53:04.614560 3812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d"
format_stamp: "Formatted at 2026-08-05 22:53:04 on dist-test-slave-7lbf"
server_key: "dcd45835708f004ea417265e228f4515"
server_key_iv: "71240c6691b6a7e615435cd035010508"
server_key_version: "encryptionkey@0"
I20260805 22:53:04.614789 3812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:53:04.621637 3812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:53:04.626498 3812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:53:04.626833 3812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:53:04.646450 3812 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:35875
I20260805 22:53:04.646507 3900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:35875 every 8 connection(s)
I20260805 22:53:04.648012 3812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:53:04.651930 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 3812
I20260805 22:53:04.652123 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:53:04.652392 3902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:53:04.652603 4208 external_mini_cluster.cc:1428] Setting key f6fe721f5aa52a648e3d0c7408a56f3f
I20260805 22:53:04.661767 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:04.654560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35415 (local address 127.4.28.62:35875)
0805 22:53:04.655447 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:04.655463 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:04.655515 (+ 52us) server_negotiation.cc:407] Connection header received
0805 22:53:04.657763 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:04.657797 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:04.658224 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:04.658614 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:04.659908 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:04.659920 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:04.659935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:53:04.659955 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:04.660053 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:53:04.660081 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:04.660285 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:04.660409 (+ 124us) server_negotiation.cc:299] Negotiation successful
0805 22:53:04.660627 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":382,"threads_started":1}
I20260805 22:53:04.666311 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:53:04.667264 3902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d: Bootstrap starting.
I20260805 22:53:04.670941 3902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d: Neither blocks nor log segments found. Creating new log.
I20260805 22:53:04.672595 3902 log.cc:824] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d: Log is configured to *not* fsync() on all Append() calls
I20260805 22:53:04.673970 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:04.670980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40195 (local address 127.4.28.62:35875)
0805 22:53:04.671206 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:04.671213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:04.671587 (+ 374us) server_negotiation.cc:407] Connection header received
0805 22:53:04.671801 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:04.671805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:04.671918 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:04.672013 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:04.673130 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:04.673134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:04.673137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 22:53:04.673154 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:04.673210 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 22:53:04.673227 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:04.673353 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:04.673554 (+ 201us) server_negotiation.cc:299] Negotiation successful
0805 22:53:04.673626 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260805 22:53:04.676456 3902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d: No bootstrap required, opened a new log
I20260805 22:53:04.684929 3902 raft_consensus.cc:348] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35875 } }
I20260805 22:53:04.685391 3902 raft_consensus.cc:374] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:53:04.685496 3902 raft_consensus.cc:729] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b650cfcf8a1f47b59c4a0f2d2e9a879d, State: Initialized, Role: FOLLOWER
I20260805 22:53:04.686213 3902 consensus_queue.cc:260] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35875 } }
I20260805 22:53:04.686492 3902 raft_consensus.cc:388] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:53:04.686636 3902 raft_consensus.cc:482] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:53:04.686815 3902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:53:04.689572 3902 raft_consensus.cc:504] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35875 } }
I20260805 22:53:04.690130 3902 leader_election.cc:302] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b650cfcf8a1f47b59c4a0f2d2e9a879d; no voters:
I20260805 22:53:04.690784 3902 leader_election.cc:288] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:53:04.690953 3916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:53:04.691648 3916 raft_consensus.cc:686] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [term 1 LEADER]: Becoming Leader. State: Replica: b650cfcf8a1f47b59c4a0f2d2e9a879d, State: Running, Role: LEADER
I20260805 22:53:04.692273 3916 consensus_queue.cc:237] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35875 } }
I20260805 22:53:04.693015 3902 sys_catalog.cc:564] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:53:04.695632 3917 sys_catalog.cc:455] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35875 } } }
I20260805 22:53:04.696170 3917 sys_catalog.cc:458] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:04.695684 3918 sys_catalog.cc:455] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b650cfcf8a1f47b59c4a0f2d2e9a879d. Latest consensus state: current_term: 1 leader_uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b650cfcf8a1f47b59c4a0f2d2e9a879d" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 35875 } } }
I20260805 22:53:04.696410 3918 sys_catalog.cc:458] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:04.697094 3929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:53:04.703157 3929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:53:04.712198 3929 catalog_manager.cc:1347] Generated new cluster ID: 706d76c4ff50491c9635c7de0f6dedf1
I20260805 22:53:04.712332 3929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:53:04.728012 3929 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:53:04.728631 3929 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:53:04.738194 3929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b650cfcf8a1f47b59c4a0f2d2e9a879d: Generated new TSK 0
I20260805 22:53:04.739240 3929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:53:04.750089 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3812
W20260805 22:53:04.751756 3895 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35415
W20260805 22:53:04.752540 4208 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3812: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260805 22:53:04.752714 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 3812
2026-08-05T22:53:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2214 ms)
[----------] 18 tests from AuthTokenIssuingTest (40740 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:53:04 dist-test-slave-7lbf krb5kdc[3950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:53:04 dist-test-slave-7lbf krb5kdc[3950](info): set up 2 sockets
Aug 05 22:53:04 dist-test-slave-7lbf krb5kdc[3950](info): commencing operation
krb5kdc: starting...
W20260805 22:53:06.004261 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.004s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:53:06 dist-test-slave-7lbf krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:53:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:53:06.385874 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:43243
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:43243,127.4.28.61:40633,127.4.28.60:44313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:53:06.755505 3969 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 22:53:06.755918 3969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:53:06.756023 3969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:53:06.756137 3969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:53:06.763783 3969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:53:06.763919 3969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:53:06.764009 3969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:53:06.764082 3969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:53:06.764171 3969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:53:06.775700 3969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:43243,127.4.28.61:40633,127.4.28.60:44313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:43243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:53:06.777282 3969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:53:06.778963 3969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:53:06.796663 3980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:06.797724 3978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:53:06.797049 3969 server_base.cc:1034] running on GCE node
W20260805 22:53:06.798287 3977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:53:06.799508 3969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:53:06.801738 3969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:53:06.803062 3969 hybrid_clock.cc:648] HybridClock initialized: now 1785970386802966 us; error 60 us; skew 500 ppm
I20260805 22:53:06.803526 3969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:53:06 dist-test-slave-7lbf krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970386, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:53:06.807126 3969 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:53:06.808740 3969 webserver.cc:460] Webserver started at http://127.4.28.62:35287/ using document root <none> and password file <none>
I20260805 22:53:06.809473 3969 fs_manager.cc:359] Metadata directory not provided
I20260805 22:53:06.809634 3969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:53:06.809939 3969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:53:06.813158 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "a7038d3e80fd4bc69ac49f2000e49cca"
format_stamp: "Formatted at 2026-08-05 22:53:06 on dist-test-slave-7lbf"
server_key: "ba6f2c0713ebc0eaf8007d68079b60be"
server_key_iv: "597a405138d7ec70e493d4332a6930e6"
server_key_version: "encryptionkey@0"
I20260805 22:53:06.814019 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "a7038d3e80fd4bc69ac49f2000e49cca"
format_stamp: "Formatted at 2026-08-05 22:53:06 on dist-test-slave-7lbf"
server_key: "ba6f2c0713ebc0eaf8007d68079b60be"
server_key_iv: "597a405138d7ec70e493d4332a6930e6"
server_key_version: "encryptionkey@0"
I20260805 22:53:06.820071 3969 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260805 22:53:06.824085 3991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:53:06.825819 3969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 22:53:06.826035 3969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "a7038d3e80fd4bc69ac49f2000e49cca"
format_stamp: "Formatted at 2026-08-05 22:53:06 on dist-test-slave-7lbf"
server_key: "ba6f2c0713ebc0eaf8007d68079b60be"
server_key_iv: "597a405138d7ec70e493d4332a6930e6"
server_key_version: "encryptionkey@0"
I20260805 22:53:06.826270 3969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:53:06.838158 3969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:53:06.843242 3969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:53:06.843547 3969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:53:06.863030 3969 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:43243
I20260805 22:53:06.863070 4066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:43243 every 8 connection(s)
I20260805 22:53:06.864694 3969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:53:06.869192 4067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:53:06.874620 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 3969
I20260805 22:53:06.874851 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:53:06.875634 4208 external_mini_cluster.cc:1428] Setting key 9045062d39c1eac0d22a57422db14a94
I20260805 22:53:06.877295 4067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:53:06 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970386, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:53:06.905722 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:06.881193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.4.28.62:43243)
0805 22:53:06.882241 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:06.882266 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:06.882852 (+ 586us) server_negotiation.cc:407] Connection header received
0805 22:53:06.885681 (+ 2829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:06.885687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:06.886004 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:06.886217 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:06.888077 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:06.889555 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:06.891308 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:06.892009 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:06.892242 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:06.897217 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:06.897229 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:06.897245 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:06.897285 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:06.900519 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:06.901736 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:06.901748 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:06.901752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:06.901971 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:06.902487 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:06.902491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:06.902493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:06.903096 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:06.903269 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:06.904036 (+ 767us) server_negotiation.cc:299] Negotiation successful
0805 22:53:06.904439 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":752,"thread_start_us":474,"threads_started":1}
I20260805 22:53:06.905684 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:06.881193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:43243 (local address 127.0.0.1:59122)
0805 22:53:06.882621 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:06.882678 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:06.885104 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:06.886275 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:06.886282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:06.887129 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:06.887760 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:06.887790 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:06.889774 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:06.889779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:06.891097 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:06.891115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:06.892341 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:06.892351 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:06.892563 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:06.893407 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:06.893446 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:06.896819 (+ 3373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:06.900741 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:06.900759 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:06.900775 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:06.901535 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:06.902128 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:06.902133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:06.902138 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:06.902372 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:06.903268 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:06.903277 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:06.903607 (+ 330us) client_negotiation.cc:769] Sending connection context
0805 22:53:06.903885 (+ 278us) client_negotiation.cc:240] Negotiation successful
0805 22:53:06.904649 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":158,"thread_start_us":900,"threads_started":1}
I20260805 22:53:06.909385 4067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:53:06.910841 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:06.910275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:43990)
0805 22:53:06.910488 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:06.910669 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260805 22:53:06.911164 3995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
I20260805 22:53:06.913033 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:06.912527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44004)
0805 22:53:06.912722 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:06.912833 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260805 22:53:06.913290 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40633: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.28.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab.
Entry for principal kudu/127.4.28.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab.
I20260805 22:53:06.962687 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.61:40633
--webserver_interface=127.4.28.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab
--principal=kudu/127.4.28.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:43243,127.4.28.61:40633,127.4.28.60:44313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:53:06.962705 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } attempt: 1
I20260805 22:53:06.964083 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:06.963570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44014)
0805 22:53:06.963816 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:06.963938 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260805 22:53:06.965274 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:06.964856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44024)
0805 22:53:06.965033 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:06.965116 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260805 22:53:06.965490 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40633: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
I20260805 22:53:07.033825 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } attempt: 2
I20260805 22:53:07.035323 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.034639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44030)
0805 22:53:07.034958 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.035090 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260805 22:53:07.036620 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.036118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44042)
0805 22:53:07.036318 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.036424 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260805 22:53:07.036849 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40633: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
I20260805 22:53:07.128185 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } attempt: 3
I20260805 22:53:07.129820 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.129108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44046)
0805 22:53:07.129411 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.129602 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260805 22:53:07.131268 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.130739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44058)
0805 22:53:07.131008 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.131096 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260805 22:53:07.131506 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40633: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
I20260805 22:53:07.274757 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } attempt: 4
I20260805 22:53:07.276269 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.275592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44072)
0805 22:53:07.275974 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.276088 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260805 22:53:07.277412 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.277009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44082)
0805 22:53:07.277187 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.277279 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260805 22:53:07.277675 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40633: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40633: connect: Connection refused (error 111)
W20260805 22:53:07.347620 4076 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 22:53:07.348050 4076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:53:07.348176 4076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:53:07.348310 4076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:53:07.356272 4076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:53:07.356410 4076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:53:07.356489 4076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:53:07.356575 4076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:53:07.356681 4076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:53:07.367499 4076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:43243,127.4.28.61:40633,127.4.28.60:44313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab
--principal=kudu/127.4.28.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.61:40633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.28.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:53:07.369063 4076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:53:07.370693 4076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:53:07.384076 4082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:07.384161 4083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:07.384188 4085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:53:07.385281 4076 server_base.cc:1034] running on GCE node
I20260805 22:53:07.386264 4076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:53:07.387964 4076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:53:07.389220 4076 hybrid_clock.cc:648] HybridClock initialized: now 1785970387389147 us; error 45 us; skew 500 ppm
I20260805 22:53:07.389663 4076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:53:07 dist-test-slave-7lbf krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:53:07.393316 4076 init.cc:377] Logged in from keytab as kudu/127.4.28.61@KRBTEST.COM (short username kudu)
I20260805 22:53:07.395203 4076 webserver.cc:460] Webserver started at http://127.4.28.61:46301/ using document root <none> and password file <none>
I20260805 22:53:07.396010 4076 fs_manager.cc:359] Metadata directory not provided
I20260805 22:53:07.396133 4076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:53:07.396493 4076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:53:07.399963 4076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/data/instance:
uuid: "d22f248ae02d4416ba7d337bd4c64395"
format_stamp: "Formatted at 2026-08-05 22:53:07 on dist-test-slave-7lbf"
server_key: "faf0c8ff996116a2df445d2351a21e68"
server_key_iv: "c8fdda6c84246cb3498de73cfd603426"
server_key_version: "encryptionkey@0"
I20260805 22:53:07.400914 4076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/wal/instance:
uuid: "d22f248ae02d4416ba7d337bd4c64395"
format_stamp: "Formatted at 2026-08-05 22:53:07 on dist-test-slave-7lbf"
server_key: "faf0c8ff996116a2df445d2351a21e68"
server_key_iv: "c8fdda6c84246cb3498de73cfd603426"
server_key_version: "encryptionkey@0"
I20260805 22:53:07.407085 4076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260805 22:53:07.411276 4092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:53:07.412985 4076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:53:07.413199 4076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/wal
uuid: "d22f248ae02d4416ba7d337bd4c64395"
format_stamp: "Formatted at 2026-08-05 22:53:07 on dist-test-slave-7lbf"
server_key: "faf0c8ff996116a2df445d2351a21e68"
server_key_iv: "c8fdda6c84246cb3498de73cfd603426"
server_key_version: "encryptionkey@0"
I20260805 22:53:07.413472 4076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:53:07.439010 4076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:53:07.443621 4076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:53:07.443929 4076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:53:07.463191 4076 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.61:40633
I20260805 22:53:07.463210 4158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.61:40633 every 8 connection(s)
I20260805 22:53:07.464615 4076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
I20260805 22:53:07.468775 4159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:53:07.472203 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4076
I20260805 22:53:07.472420 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-1/wal/instance
I20260805 22:53:07.472864 4208 external_mini_cluster.cc:1428] Setting key d0dae2d5b34b3c88f56e77097b883442
I20260805 22:53:07.476085 4159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:53:07 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:53:07.501044 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.479079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.4.28.62:43243)
0805 22:53:07.479591 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:07.479596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:07.480990 (+ 1394us) server_negotiation.cc:407] Connection header received
0805 22:53:07.483269 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:07.483276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:07.483389 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:07.483507 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:07.485412 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.486227 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.487973 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.488446 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.488582 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:07.493391 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:07.493398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:07.493402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:07.493463 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:07.496700 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:07.497961 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:07.497967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:07.497969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:07.498144 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:07.498832 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:07.498837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:07.498840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:07.499248 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:07.499419 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:07.500491 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0805 22:53:07.500714 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":279,"threads_started":1}
I20260805 22:53:07.502554 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.479749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:43243 (local address 127.0.0.1:59136)
0805 22:53:07.480742 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.480832 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:07.482920 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:07.483832 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:07.483843 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:07.484461 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:07.485129 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:07.485150 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.486435 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.486439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:07.487793 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:07.487810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.488717 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.488726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:07.489006 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:07.489841 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:07.489883 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:07.493015 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:07.496958 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:07.496970 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:07.496993 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:07.497776 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:07.498321 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:07.498327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:07.498330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:07.498653 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:07.499438 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:07.499449 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:07.499988 (+ 539us) client_negotiation.cc:769] Sending connection context
0805 22:53:07.500329 (+ 341us) client_negotiation.cc:240] Negotiation successful
0805 22:53:07.500965 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":422,"threads_started":1}
I20260805 22:53:07.504837 4159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.28.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.60@KRBTEST.COM" created.
Aug 05 22:53:07 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
I20260805 22:53:07.521449 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.505746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44092)
0805 22:53:07.505962 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.505991 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:07.506117 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:07.507195 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:07.507201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:07.507713 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:07.507920 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:07.507933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.509408 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.509414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:07.510847 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:07.510864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.511821 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.511826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:07.511956 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:07.512509 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:07.512527 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:07.514870 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:07.518781 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:07.518786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:07.518791 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:07.519204 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:07.519600 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:07.519605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:07.519609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:07.519758 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:07.520407 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:07.520416 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:07.520581 (+ 165us) client_negotiation.cc:769] Sending connection context
0805 22:53:07.520697 (+ 116us) client_negotiation.cc:240] Negotiation successful
0805 22:53:07.521031 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260805 22:53:07.521508 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.506009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.4.28.61:40633)
0805 22:53:07.506717 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:07.506725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:07.506754 (+ 29us) server_negotiation.cc:407] Connection header received
0805 22:53:07.506802 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:07.506808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:07.507010 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:07.507139 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:07.508103 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.509230 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.511002 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.511596 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.511699 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:07.515138 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:07.515154 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:07.515172 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:07.515208 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:07.518602 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:07.519354 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:07.519362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:07.519364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:07.519490 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:07.519868 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:07.519873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:07.519875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:07.520268 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:07.520499 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:07.520904 (+ 405us) server_negotiation.cc:299] Negotiation successful
0805 22:53:07.521150 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":392,"threads_started":1}
I20260805 22:53:07.524871 4159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:53:07.526130 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.525626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:32886)
0805 22:53:07.525819 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.525961 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260805 22:53:07.526401 4097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
I20260805 22:53:07.528106 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.527730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:32892)
0805 22:53:07.527890 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.527980 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260805 22:53:07.528332 4159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab.
Entry for principal kudu/127.4.28.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab.
I20260805 22:53:07.557425 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.60:44313
--webserver_interface=127.4.28.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab
--principal=kudu/127.4.28.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:43243,127.4.28.61:40633,127.4.28.60:44313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:53:07.576967 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } attempt: 5
I20260805 22:53:07.577665 4159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 1
I20260805 22:53:07.578804 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.578262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:32904)
0805 22:53:07.578520 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.578623 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260805 22:53:07.580026 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.579611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:32920)
0805 22:53:07.579792 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.579887 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260805 22:53:07.580425 4159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Aug 05 22:53:07 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970386, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
I20260805 22:53:07.592767 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.578315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.4.28.61:40633)
0805 22:53:07.578511 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:07.578516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:07.578537 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:53:07.578598 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:07.578603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:07.578711 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:07.578824 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:07.579848 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.581066 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.582393 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.583006 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.583133 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:07.586912 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:07.586918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:07.586922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:07.586965 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:07.589840 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:07.590815 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:07.590819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:07.590822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:07.590962 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:07.591381 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:07.591384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:07.591386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:07.591699 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:07.591821 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:07.592306 (+ 485us) server_negotiation.cc:299] Negotiation successful
0805 22:53:07.592508 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260805 22:53:07.593109 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.577854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:44108)
0805 22:53:07.578148 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.578173 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:07.578312 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:07.578913 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:07.578918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:07.579463 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:07.579679 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:07.579695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.581233 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.581238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:07.582217 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:07.582239 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:07.583192 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:07.583198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:07.583403 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:07.584063 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:07.584083 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:07.586627 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:07.590037 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:07.590045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:07.590049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:07.590678 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:07.591081 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:07.591086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:07.591090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:07.591277 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:07.591821 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:07.591826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:07.592031 (+ 205us) client_negotiation.cc:769] Sending connection context
0805 22:53:07.592203 (+ 172us) client_negotiation.cc:240] Negotiation successful
0805 22:53:07.592582 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260805 22:53:07.594254 4067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:53:07.595436 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.595016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:32930)
0805 22:53:07.595194 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.595278 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260805 22:53:07.596460 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.596081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:32942)
0805 22:53:07.596237 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.596347 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260805 22:53:07.596755 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
I20260805 22:53:07.648038 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 1
I20260805 22:53:07.648658 4159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 2
I20260805 22:53:07.649427 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.648820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40866)
0805 22:53:07.649122 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.649219 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260805 22:53:07.649873 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.649289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40868)
0805 22:53:07.649581 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.649689 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260805 22:53:07.650810 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.650310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40876)
0805 22:53:07.650572 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.650663 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260805 22:53:07.650995 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.650611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40878)
0805 22:53:07.650773 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.650851 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260805 22:53:07.651073 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
W20260805 22:53:07.651314 4159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
I20260805 22:53:07.719308 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 2
I20260805 22:53:07.720652 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.720084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40892)
0805 22:53:07.720361 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.720467 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260805 22:53:07.721881 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.721444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40908)
0805 22:53:07.721643 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.721723 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260805 22:53:07.722141 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
I20260805 22:53:07.742571 4159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 3
I20260805 22:53:07.743835 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.743322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40922)
0805 22:53:07.743566 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.743661 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260805 22:53:07.744980 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.744498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40932)
0805 22:53:07.744762 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.744848 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260805 22:53:07.745223 4159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
I20260805 22:53:07.828383 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 3
I20260805 22:53:07.829777 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.829209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40934)
0805 22:53:07.829434 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.829586 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260805 22:53:07.830904 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.830494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40948)
0805 22:53:07.830689 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.830778 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260805 22:53:07.831218 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
I20260805 22:53:07.888485 4159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 4
I20260805 22:53:07.889786 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.889202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40956)
0805 22:53:07.889464 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.889600 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260805 22:53:07.890934 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:07.890562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40966)
0805 22:53:07.890727 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:07.890809 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260805 22:53:07.891147 4159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
W20260805 22:53:07.942569 4167 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 22:53:07.943023 4167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:53:07.943152 4167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:53:07.943288 4167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:53:07.951376 4167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:53:07.951536 4167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:53:07.951635 4167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:53:07.951714 4167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:53:07.951825 4167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:53:07.962842 4167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:43243,127.4.28.61:40633,127.4.28.60:44313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab
--principal=kudu/127.4.28.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.60:44313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.28.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:53:07.964433 4167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:53:07.966133 4167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:53:07.980043 4177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:07.980111 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:07.980077 4179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:53:07.981252 4167 server_base.cc:1034] running on GCE node
I20260805 22:53:07.982227 4167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:53:07.983824 4167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:53:07.985100 4167 hybrid_clock.cc:648] HybridClock initialized: now 1785970387985043 us; error 50 us; skew 500 ppm
I20260805 22:53:07.985488 4167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:53:07 dist-test-slave-7lbf krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:53:07.989324 4167 init.cc:377] Logged in from keytab as kudu/127.4.28.60@KRBTEST.COM (short username kudu)
I20260805 22:53:07.991074 4167 webserver.cc:460] Webserver started at http://127.4.28.60:36921/ using document root <none> and password file <none>
I20260805 22:53:07.991854 4167 fs_manager.cc:359] Metadata directory not provided
I20260805 22:53:07.991976 4167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:53:07.992328 4167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:53:07.995735 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/data/instance:
uuid: "902d31756cbb42948823ea96c700eb00"
format_stamp: "Formatted at 2026-08-05 22:53:07 on dist-test-slave-7lbf"
server_key: "f0b5a282331734d0be0b149db1a9c814"
server_key_iv: "78f286ba7f66630bf32ace9a1c2eaa44"
server_key_version: "encryptionkey@0"
I20260805 22:53:07.996660 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/wal/instance:
uuid: "902d31756cbb42948823ea96c700eb00"
format_stamp: "Formatted at 2026-08-05 22:53:07 on dist-test-slave-7lbf"
server_key: "f0b5a282331734d0be0b149db1a9c814"
server_key_iv: "78f286ba7f66630bf32ace9a1c2eaa44"
server_key_version: "encryptionkey@0"
I20260805 22:53:08.002801 4167 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260805 22:53:08.006955 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:53:08.008486 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 4
I20260805 22:53:08.008603 4167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:53:08.008790 4167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/wal
uuid: "902d31756cbb42948823ea96c700eb00"
format_stamp: "Formatted at 2026-08-05 22:53:07 on dist-test-slave-7lbf"
server_key: "f0b5a282331734d0be0b149db1a9c814"
server_key_iv: "78f286ba7f66630bf32ace9a1c2eaa44"
server_key_version: "encryptionkey@0"
I20260805 22:53:08.009115 4167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:53:08.009732 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.009219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40976)
0805 22:53:08.009446 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:08.009586 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260805 22:53:08.010982 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.010596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:40980)
0805 22:53:08.010751 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:08.010830 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260805 22:53:08.011190 4067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:44313: Network error: Client connection negotiation failed: client connection to 127.4.28.60:44313: connect: Connection refused (error 111)
I20260805 22:53:08.023736 4167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:53:08.028645 4167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:53:08.028931 4167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:53:08.048887 4167 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.60:44313
I20260805 22:53:08.048890 4267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.60:44313 every 8 connection(s)
I20260805 22:53:08.050454 4167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
I20260805 22:53:08.054674 4268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:53:08.059137 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4167
I20260805 22:53:08.059366 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785970002413767-4208-0/minicluster-data/master-2/wal/instance
I20260805 22:53:08.059892 4208 external_mini_cluster.cc:1428] Setting key da9f88a8193d1efa94213eb79b83e23e
I20260805 22:53:08.062356 4268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:53:08 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
Aug 05 22:53:08 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
I20260805 22:53:08.082397 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.065867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.4.28.62:43243)
0805 22:53:08.067243 (+ 1376us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.067247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.067264 (+ 17us) server_negotiation.cc:407] Connection header received
0805 22:53:08.067315 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.067318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.067387 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.067492 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.068931 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.069748 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.071272 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.071703 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.071825 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.075784 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.075790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.075793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.075833 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.078740 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.079658 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.079662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.079665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.079862 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.080341 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.080346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.080348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.080660 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.080807 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.081698 (+ 891us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.081972 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":1145,"threads_started":1}
I20260805 22:53:08.083506 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.062187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.4.28.60:44313)
0805 22:53:08.063168 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.063185 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.063242 (+ 57us) server_negotiation.cc:407] Connection header received
0805 22:53:08.065559 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.065590 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.066147 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.067241 (+ 1094us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.068494 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.069769 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.071150 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.071726 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.071909 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.076209 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.076220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.076235 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.076271 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.079274 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.080039 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.080048 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.080050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.080264 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.080673 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.080678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.080680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.081304 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.081686 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.081994 (+ 308us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.082335 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":700,"spinlock_wait_cycles":19712,"thread_start_us":432,"threads_started":1}
I20260805 22:53:08.083683 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.065817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:43243 (local address 127.0.0.1:60500)
0805 22:53:08.066616 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:08.066661 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:08.066813 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:08.067606 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:08.067614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:08.068259 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:08.068727 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.068744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.069910 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.069914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.071094 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.071108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.071961 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.071969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.072195 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.072926 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:08.072959 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:08.075531 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:08.078930 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.078947 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.078962 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.079505 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.080000 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.080005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.080009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.080216 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.080799 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:08.080809 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:08.081277 (+ 468us) client_negotiation.cc:769] Sending connection context
0805 22:53:08.081595 (+ 318us) client_negotiation.cc:240] Negotiation successful
0805 22:53:08.082185 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":49,"thread_start_us":351,"threads_started":1}
I20260805 22:53:08.086217 4268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:53:08 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
Aug 05 22:53:08 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:53:08.102252 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.087105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40633 (local address 127.0.0.1:56944)
0805 22:53:08.087750 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:08.087769 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:08.087883 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:08.088241 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:08.088245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:08.088629 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:08.088782 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.088791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.090022 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.090037 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.091139 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.091154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.092143 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.092149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.092372 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.093101 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:08.093121 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:08.095539 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:08.099245 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.099252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.099256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.099862 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.100332 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.100337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.100342 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.100597 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.101169 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:08.101174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:08.101401 (+ 227us) client_negotiation.cc:769] Sending connection context
0805 22:53:08.101601 (+ 200us) client_negotiation.cc:240] Negotiation successful
0805 22:53:08.101917 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526}
I20260805 22:53:08.102316 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.087162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.4.28.61:40633)
0805 22:53:08.087385 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.087391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.087845 (+ 454us) server_negotiation.cc:407] Connection header received
0805 22:53:08.088013 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.088018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.088101 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.088221 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.088940 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.089861 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.091315 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.091930 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.092045 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.095869 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.095878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.095883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.095937 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.098981 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.100005 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.100009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.100011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.100194 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.100702 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.100705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.100708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.100992 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.101157 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.101711 (+ 554us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.101962 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260805 22:53:08.103587 4268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:53:08.104120 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.087345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.4.28.62:43243)
0805 22:53:08.087561 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.087567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.087588 (+ 21us) server_negotiation.cc:407] Connection header received
0805 22:53:08.087657 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.087662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.087755 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.087852 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.089429 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.090326 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.091834 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.092390 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.092525 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.097345 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.097352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.097355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.097398 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.100767 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.101752 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.101759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.101763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.101945 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.102480 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.102485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.102488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.102882 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.103204 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.103576 (+ 372us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.103797 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 05 22:53:08 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
Aug 05 22:53:08 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
I20260805 22:53:08.118865 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.104395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:41004)
0805 22:53:08.104591 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:08.104612 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:08.104725 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:08.105099 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:08.105103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:08.105558 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:08.105712 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.105722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.106982 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.106987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.107934 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.107952 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.108671 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.108678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.108877 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.109483 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:08.109502 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:08.112141 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:08.116038 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.116043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.116046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.116587 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.116956 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.116961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.116964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.117167 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.117828 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:08.117833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:08.118040 (+ 207us) client_negotiation.cc:769] Sending connection context
0805 22:53:08.118218 (+ 178us) client_negotiation.cc:240] Negotiation successful
0805 22:53:08.118571 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260805 22:53:08.119199 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.104685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.4.28.60:44313)
0805 22:53:08.104842 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.104847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.104862 (+ 15us) server_negotiation.cc:407] Connection header received
0805 22:53:08.104919 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.104923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.104976 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.105062 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.105847 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.106841 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.108096 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.108506 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.108615 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.112376 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.112384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.112389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.112431 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.115856 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.116720 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.116724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.116727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.116855 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.117274 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.117278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.117280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.117702 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.117880 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.118369 (+ 489us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.118759 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":51}
I20260805 22:53:08.123621 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.105975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.4.28.61:40633)
0805 22:53:08.106161 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.106166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.106183 (+ 17us) server_negotiation.cc:407] Connection header received
0805 22:53:08.106344 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.106348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.106404 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.106478 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.107888 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.108780 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.110158 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.110827 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.110934 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.116443 (+ 5509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.116451 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.116457 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.116509 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.120105 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.121087 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.121094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.121097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.121273 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.121897 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.121920 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.121924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.122338 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.122633 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.123037 (+ 404us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.123303 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260805 22:53:08.125447 4268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00: Bootstrap starting.
I20260805 22:53:08.129329 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:53:08.129590 4268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00: Neither blocks nor log segments found. Creating new log.
I20260805 22:53:08.130930 4268 log.cc:824] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00: Log is configured to *not* fsync() on all Append() calls
I20260805 22:53:08.134893 4268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00: No bootstrap required, opened a new log
I20260805 22:53:08.132829 4033 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a7038d3e80fd4bc69ac49f2000e49cca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514
I20260805 22:53:08.146356 4268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } }
I20260805 22:53:08.146871 4268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902d31756cbb42948823ea96c700eb00, State: Initialized, Role: FOLLOWER
I20260805 22:53:08.147452 4268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } }
I20260805 22:53:08.148574 4279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.148939 4279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:08.150168 4268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [sys.catalog]: configured and running, proceeding with master startup.
W20260805 22:53:08.160378 4293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 22:53:08.160544 4293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 22:53:08.158353 4167 master_runner.cc:186] Error getting master registration for 127.4.28.62:43243: OK, instance_id { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" instance_seqno: 1785970386857469 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 22:53:08.169768 4167 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260805 22:53:08.190375 4159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 5
Aug 05 22:53:08 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
I20260805 22:53:08.209944 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.191270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:41018)
0805 22:53:08.191510 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:08.191535 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:08.191687 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:08.192097 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:08.192102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:08.192696 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:08.192936 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.192951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.194566 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.194572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.196056 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.196113 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.197396 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.197404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.197671 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.198433 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:08.198460 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:08.201678 (+ 3218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:08.206585 (+ 4907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.206592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.206597 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.207223 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.207721 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.207727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.207732 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.207937 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.208782 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:08.208787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:08.208978 (+ 191us) client_negotiation.cc:769] Sending connection context
0805 22:53:08.209148 (+ 170us) client_negotiation.cc:240] Negotiation successful
0805 22:53:08.209496 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260805 22:53:08.210219 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.191604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.4.28.60:44313)
0805 22:53:08.191772 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.191778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.191797 (+ 19us) server_negotiation.cc:407] Connection header received
0805 22:53:08.191860 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.191865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.191928 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.192030 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.193133 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.194355 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.196324 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.197172 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.197306 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.202008 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.202018 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.202024 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.202082 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.206349 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.207379 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.207386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.207391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.207581 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.208065 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.208071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.208075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.208644 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.208821 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.209301 (+ 480us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.209612 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 22:53:08.215430 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395: Bootstrap starting.
I20260805 22:53:08.219172 4159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395: Neither blocks nor log segments found. Creating new log.
I20260805 22:53:08.220355 4159 log.cc:824] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395: Log is configured to *not* fsync() on all Append() calls
I20260805 22:53:08.223413 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395: No bootstrap required, opened a new log
I20260805 22:53:08.231555 4159 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } }
I20260805 22:53:08.231958 4159 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22f248ae02d4416ba7d337bd4c64395, State: Initialized, Role: FOLLOWER
I20260805 22:53:08.232506 4159 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } }
I20260805 22:53:08.233606 4296 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.234066 4296 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:08.234921 4159 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [sys.catalog]: configured and running, proceeding with master startup.
W20260805 22:53:08.245978 4312 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 22:53:08.246170 4312 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 22:53:08.242892 4076 master_runner.cc:186] Error getting master registration for 127.4.28.62:43243: OK, instance_id { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" instance_seqno: 1785970386857469 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 22:53:08.256306 4076 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260805 22:53:08.281741 4204 tablet.cc:2369] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:53:08.288444 4067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } attempt: 5
Aug 05 22:53:08 dist-test-slave-7lbf krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970386, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
I20260805 22:53:08.307914 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.289298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:44313 (local address 127.0.0.1:41024)
0805 22:53:08.289710 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:08.289734 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:08.289885 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:08.290382 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:08.290388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:08.291026 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:08.291314 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.291331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.292951 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.292958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.294492 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:08.294514 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.295715 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.295723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:08.295975 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.296753 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:08.296781 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:08.299935 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:08.304546 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.304552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.304557 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.305149 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.305752 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:08.305758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:08.305762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:08.305994 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:08.306795 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:08.306801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:08.306999 (+ 198us) client_negotiation.cc:769] Sending connection context
0805 22:53:08.307159 (+ 160us) client_negotiation.cc:240] Negotiation successful
0805 22:53:08.307484 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260805 22:53:08.309029 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.289581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.4.28.60:44313)
0805 22:53:08.289796 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.289802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.289830 (+ 28us) server_negotiation.cc:407] Connection header received
0805 22:53:08.290053 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.290060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.290167 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.290295 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.291527 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.292745 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.294718 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.295487 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.295643 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.300256 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.300265 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.300270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.300319 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.304273 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.305332 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.305340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.305343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.305585 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.306166 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.306172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.306176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.306563 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.306724 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.308375 (+ 1651us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.308655 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260805 22:53:08.316092 4067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca: Bootstrap starting.
I20260805 22:53:08.321031 4067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca: Neither blocks nor log segments found. Creating new log.
I20260805 22:53:08.322682 4067 log.cc:824] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca: Log is configured to *not* fsync() on all Append() calls
I20260805 22:53:08.327422 4067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca: No bootstrap required, opened a new log
I20260805 22:53:08.336392 4067 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } }
I20260805 22:53:08.336966 4067 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7038d3e80fd4bc69ac49f2000e49cca, State: Initialized, Role: FOLLOWER
I20260805 22:53:08.337764 4067 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } }
I20260805 22:53:08.339347 4316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.340113 4316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:08.340632 4067 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:53:08.340827 4033 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:53:08.341063 4033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:53:08.344671 4033 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } }
W20260805 22:53:08.347119 3997 tablet.cc:2369] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:53:08.347709 4130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d22f248ae02d4416ba7d337bd4c64395"
I20260805 22:53:08.348255 4130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:53:08.352151 4130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7038d3e80fd4bc69ac49f2000e49cca in term 1.
I20260805 22:53:08.353169 3995 leader_election.cc:302] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7038d3e80fd4bc69ac49f2000e49cca, d22f248ae02d4416ba7d337bd4c64395; no voters:
I20260805 22:53:08.355078 4238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "902d31756cbb42948823ea96c700eb00"
I20260805 22:53:08.355646 4238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:53:08.359432 4238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7038d3e80fd4bc69ac49f2000e49cca in term 1.
I20260805 22:53:08.360404 4033 leader_election.cc:288] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [CANDIDATE]: Term 1 election: Requested vote from peers d22f248ae02d4416ba7d337bd4c64395 (127.4.28.61:40633), 902d31756cbb42948823ea96c700eb00 (127.4.28.60:44313)
I20260805 22:53:08.367766 4316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:53:08.378190 4316 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [term 1 LEADER]: Becoming Leader. State: Replica: a7038d3e80fd4bc69ac49f2000e49cca, State: Running, Role: LEADER
I20260805 22:53:08.380376 4316 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } }
I20260805 22:53:08.399039 4325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7038d3e80fd4bc69ac49f2000e49cca. Latest consensus state: current_term: 1 leader_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.399698 4325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:08.420037 4336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:53:08.438187 4336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260805 22:53:08.446959 4099 tablet.cc:2369] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:53:08.449579 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.403714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.4.28.62:43243)
0805 22:53:08.403954 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.403962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.414809 (+ 10847us) server_negotiation.cc:407] Connection header received
0805 22:53:08.415241 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.415249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.415354 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.415436 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.420027 (+ 4591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.421244 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.423202 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.423909 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.424008 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.427867 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.427875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.427880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.427924 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.431931 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.436788 (+ 4857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.436796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.436800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.437017 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.437601 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.437607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.437611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.438053 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.438449 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.448789 (+ 10340us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.449105 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 22:53:08.452214 4238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [term 1 FOLLOWER]: Refusing update from remote peer a7038d3e80fd4bc69ac49f2000e49cca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:53:08.453678 4130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [term 1 FOLLOWER]: Refusing update from remote peer a7038d3e80fd4bc69ac49f2000e49cca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:53:08.455276 4316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:53:08.456110 4316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [LEADER]: Connected to new peer: Peer: permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:53:08.475600 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.412081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.4.28.61:40633)
0805 22:53:08.412297 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.412304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.429915 (+ 17611us) server_negotiation.cc:407] Connection header received
0805 22:53:08.431180 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.431188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.431300 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.431390 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.445437 (+ 14047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.446697 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.451022 (+ 4325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.462257 (+ 11235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.462428 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.466144 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.466156 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.466161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.466218 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.470342 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.471373 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.471379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.471383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.471661 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.472280 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.472286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.472289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.472738 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.473002 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.474671 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.475069 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 22:53:08.477845 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:08.430629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.4.28.60:44313)
0805 22:53:08.430850 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.430857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.436469 (+ 5612us) server_negotiation.cc:407] Connection header received
0805 22:53:08.437226 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.437234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.437372 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.437463 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.445458 (+ 7995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.446619 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.452644 (+ 6025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:08.453233 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:08.453379 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:08.457620 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:08.457631 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:08.457638 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:08.457691 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:08.464293 (+ 6602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.469954 (+ 5661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.469962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.469966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.470171 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:08.475046 (+ 4875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:08.475053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:08.475056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:08.475512 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:08.475782 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:08.477104 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0805 22:53:08.477383 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260805 22:53:08.492820 4279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7038d3e80fd4bc69ac49f2000e49cca. Latest consensus state: current_term: 1 leader_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.493237 4279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:08.498896 4349 mvcc.cc:205] Tried to move back new op lower bound from 7315334711085338624 to 7315334710824505344. Current Snapshot: MvccSnapshot[applied={T|T < 7315334711085338624}]
I20260805 22:53:08.503713 4296 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7038d3e80fd4bc69ac49f2000e49cca. Latest consensus state: current_term: 1 leader_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.504068 4296 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:08.507061 4348 mvcc.cc:205] Tried to move back new op lower bound from 7315334711085338624 to 7315334710824505344. Current Snapshot: MvccSnapshot[applied={T|T < 7315334711085338624}]
I20260805 22:53:08.511457 4296 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.511741 4296 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:08.524351 4316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.524832 4316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:08.527271 4336 catalog_manager.cc:1347] Generated new cluster ID: a588eba343f34a4094a1f0ac3b441019
I20260805 22:53:08.527439 4336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:53:08.533828 4325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.534309 4325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:08.538085 4279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7038d3e80fd4bc69ac49f2000e49cca" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 43243 } } peers { permanent_uuid: "d22f248ae02d4416ba7d337bd4c64395" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40633 } } peers { permanent_uuid: "902d31756cbb42948823ea96c700eb00" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 44313 } } }
I20260805 22:53:08.538460 4279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:08.558560 4336 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:53:08.562095 4336 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:53:08.583256 4336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7038d3e80fd4bc69ac49f2000e49cca: Generated new TSK 0
I20260805 22:53:08.586066 4336 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 22:53:08.736287 4277 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.724490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.4.28.62:43243)
0805 22:53:08.724788 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.724797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.734203 (+ 9406us) server_negotiation.cc:407] Connection header received
0805 22:53:08.734545 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.734551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.734644 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.734722 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.736012 (+ 1290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60534: BlockingRecv error: recv got EOF from 127.0.0.1:60534 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260805 22:53:08.738826 4271 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.735635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.4.28.60:44313)
0805 22:53:08.735832 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.735839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.737103 (+ 1264us) server_negotiation.cc:407] Connection header received
0805 22:53:08.737356 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.737361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.737447 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.737543 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.738578 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41046: BlockingRecv error: recv got EOF from 127.0.0.1:41046 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260805 22:53:08.742694 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.732994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.4.28.61:40633)
0805 22:53:08.733224 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.733232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.734897 (+ 1665us) server_negotiation.cc:407] Connection header received
0805 22:53:08.735099 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.735106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.735205 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.735305 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.742394 (+ 7089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56972: BlockingRecv error: recv got EOF from 127.0.0.1:56972 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260805 22:53:08.792331 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.782680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.4.28.61:40633)
0805 22:53:08.782873 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.782881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.789995 (+ 7114us) server_negotiation.cc:407] Connection header received
0805 22:53:08.790257 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.790264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.790636 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.790768 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.792004 (+ 1236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56980: BlockingRecv error: recv got EOF from 127.0.0.1:56980 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260805 22:53:08.812094 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.806164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.4.28.61:40633)
0805 22:53:08.806368 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.806375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.810307 (+ 3932us) server_negotiation.cc:407] Connection header received
0805 22:53:08.810564 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.810571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.810694 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.810854 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.811831 (+ 977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56982: BlockingRecv error: recv got EOF from 127.0.0.1:56982 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260805 22:53:08.821192 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.818774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.4.28.61:40633)
0805 22:53:08.818994 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.819001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.819530 (+ 529us) server_negotiation.cc:407] Connection header received
0805 22:53:08.819741 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.819748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.819874 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.819998 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.820915 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56986: BlockingRecv error: recv got EOF from 127.0.0.1:56986 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260805 22:53:08.831212 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.828683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.4.28.61:40633)
0805 22:53:08.828930 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.828937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.829498 (+ 561us) server_negotiation.cc:407] Connection header received
0805 22:53:08.829745 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.829753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.829867 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.829988 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.830945 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57002: BlockingRecv error: recv got EOF from 127.0.0.1:57002 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260805 22:53:08.847762 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.844674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.4.28.61:40633)
0805 22:53:08.845053 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.845082 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.845851 (+ 769us) server_negotiation.cc:407] Connection header received
0805 22:53:08.846217 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.846223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.846321 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.846409 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.847515 (+ 1106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57006: BlockingRecv error: recv got EOF from 127.0.0.1:57006 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260805 22:53:08.884688 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.881450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.4.28.61:40633)
0805 22:53:08.881981 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.881989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.882633 (+ 644us) server_negotiation.cc:407] Connection header received
0805 22:53:08.883042 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.883049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.883160 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.883252 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.884422 (+ 1170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57018: BlockingRecv error: recv got EOF from 127.0.0.1:57018 (error 108)
Metrics: {"server-negotiator.queue_time_us":358}
W20260805 22:53:08.924768 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.922539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.4.28.61:40633)
0805 22:53:08.922759 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.922766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.923169 (+ 403us) server_negotiation.cc:407] Connection header received
0805 22:53:08.923396 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.923400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.923479 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.923570 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.924488 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57032: BlockingRecv error: recv got EOF from 127.0.0.1:57032 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260805 22:53:08.995545 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:08.993352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.4.28.61:40633)
0805 22:53:08.993622 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:08.993629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:08.994047 (+ 418us) server_negotiation.cc:407] Connection header received
0805 22:53:08.994237 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:08.994243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:08.994357 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:08.994503 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:08.995303 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57040: BlockingRecv error: recv got EOF from 127.0.0.1:57040 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260805 22:53:09.133489 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:09.131112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.4.28.61:40633)
0805 22:53:09.131393 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:09.131402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:09.131879 (+ 477us) server_negotiation.cc:407] Connection header received
0805 22:53:09.132110 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:09.132115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:09.132200 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:09.132299 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:09.133221 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57056: BlockingRecv error: recv got EOF from 127.0.0.1:57056 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260805 22:53:09.162909 4293 catalog_manager.cc:1259] Loaded cluster ID: a588eba343f34a4094a1f0ac3b441019
I20260805 22:53:09.163028 4293 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00: loading cluster ID for follower catalog manager: success
I20260805 22:53:09.165902 4293 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00: acquiring CA information for follower catalog manager: success
I20260805 22:53:09.167938 4293 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 902d31756cbb42948823ea96c700eb00: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 22:53:09.248255 4312 catalog_manager.cc:1259] Loaded cluster ID: a588eba343f34a4094a1f0ac3b441019
I20260805 22:53:09.248373 4312 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395: loading cluster ID for follower catalog manager: success
I20260805 22:53:09.251195 4312 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395: acquiring CA information for follower catalog manager: success
I20260805 22:53:09.253293 4312 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d22f248ae02d4416ba7d337bd4c64395: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 22:53:09.400220 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:09.393849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.4.28.61:40633)
0805 22:53:09.394102 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:09.394110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:09.394419 (+ 309us) server_negotiation.cc:407] Connection header received
0805 22:53:09.394593 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:09.394597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:09.394777 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:09.394888 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:09.395404 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.396223 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.397720 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.398175 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.398288 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:09.398952 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:09.399540 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:09.399673 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:09.399756 (+ 83us) server_negotiation.cc:299] Negotiation successful
0805 22:53:09.399907 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260805 22:53:09.413496 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:09.406889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.4.28.60:44313)
0805 22:53:09.407836 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:09.407841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:09.407859 (+ 18us) server_negotiation.cc:407] Connection header received
0805 22:53:09.407965 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:09.407969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:09.408174 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:09.408324 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:09.408837 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.409686 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.411111 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.411519 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.411623 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:09.412229 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:09.412880 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:09.413000 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:09.413145 (+ 145us) server_negotiation.cc:299] Negotiation successful
0805 22:53:09.413223 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":704,"threads_started":1}
I20260805 22:53:09.429760 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:09.420545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.4.28.61:40633)
0805 22:53:09.420762 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:09.420771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:09.422290 (+ 1519us) server_negotiation.cc:407] Connection header received
0805 22:53:09.422665 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:09.422671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:09.422754 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:09.422854 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:09.424281 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.425310 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.427552 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.428196 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.428315 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:09.428896 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:09.429082 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:09.429168 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:09.429367 (+ 199us) server_negotiation.cc:299] Negotiation successful
0805 22:53:09.429455 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260805 22:53:09.429809 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:09.421739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.4.28.60:44313)
0805 22:53:09.421961 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:09.421968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:09.422861 (+ 893us) server_negotiation.cc:407] Connection header received
0805 22:53:09.423124 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:09.423131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:09.423229 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:09.423347 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:09.424096 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.424932 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.426852 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.427268 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.427383 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:09.428916 (+ 1533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:09.429038 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:09.429130 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:09.429301 (+ 171us) server_negotiation.cc:299] Negotiation successful
0805 22:53:09.429400 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260805 22:53:09.431393 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:09.420417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.4.28.62:43243)
0805 22:53:09.422382 (+ 1965us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:09.422388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:09.422411 (+ 23us) server_negotiation.cc:407] Connection header received
0805 22:53:09.422548 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:09.422555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:09.422811 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:09.423022 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:09.423709 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.424823 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.427418 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:09.428023 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:09.428151 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:09.428878 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:09.429426 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:09.429577 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:09.430212 (+ 635us) server_negotiation.cc:299] Negotiation successful
0805 22:53:09.430306 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1840,"thread_start_us":1728,"threads_started":1}
I20260805 22:53:09.437059 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 3969
I20260805 22:53:09.438594 4061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:53:09.561805 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 3969
W20260805 22:53:09.589995 4095 connection.cc:535] server connection from 127.0.0.1:44108 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260805 22:53:09.590739 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4076
I20260805 22:53:09.592106 4152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:53:09.702359 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4076
I20260805 22:53:09.728291 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4167
I20260805 22:53:09.729642 4262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:53:09.838800 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4167
2026-08-05T22:53:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5082 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 22:53:09 dist-test-slave-7lbf krb5kdc[4475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 22:53:09 dist-test-slave-7lbf krb5kdc[4475](info): set up 2 sockets
Aug 05 22:53:09 dist-test-slave-7lbf krb5kdc[4475](info): commencing operation
krb5kdc: starting...
W20260805 22:53:11.105412 4208 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.001s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/test-user.keytab.
Aug 05 22:53:11 dist-test-slave-7lbf krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T22:53:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T22:53:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.28.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
Entry for principal kudu/127.4.28.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab.
I20260805 22:53:11.488968 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45061
--webserver_interface=127.4.28.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45061,127.4.28.61:40367,127.4.28.60:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 22:53:11.867542 4494 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 22:53:11.867959 4494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:53:11.868039 4494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:53:11.868124 4494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:53:11.875766 4494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:53:11.875895 4494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:53:11.875958 4494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:53:11.876010 4494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:53:11.876075 4494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:53:11.887117 4494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45061,127.4.28.61:40367,127.4.28.60:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.62.keytab
--principal=kudu/127.4.28.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.62:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.28.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:53:11.888661 4494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:53:11.890239 4494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:53:11.903087 4502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:11.903160 4500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:11.903060 4499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:53:11.907339 4494 server_base.cc:1034] running on GCE node
I20260805 22:53:11.908368 4494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:53:11.910045 4494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:53:11.911324 4494 hybrid_clock.cc:648] HybridClock initialized: now 1785970391911247 us; error 41 us; skew 500 ppm
I20260805 22:53:11.911698 4494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:53:11 dist-test-slave-7lbf krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970391, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:53:11.915194 4494 init.cc:377] Logged in from keytab as kudu/127.4.28.62@KRBTEST.COM (short username kudu)
I20260805 22:53:11.916971 4494 webserver.cc:460] Webserver started at http://127.4.28.62:32993/ using document root <none> and password file <none>
I20260805 22:53:11.917835 4494 fs_manager.cc:359] Metadata directory not provided
I20260805 22:53:11.917963 4494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:53:11.918289 4494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:53:11.921654 4494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/data/instance:
uuid: "037d3027adb3427d9953543c7c742bc9"
format_stamp: "Formatted at 2026-08-05 22:53:11 on dist-test-slave-7lbf"
server_key: "e59c60463836cc951fc9624233d98301"
server_key_iv: "4359d3043a97ab0a6861d026143e4559"
server_key_version: "encryptionkey@0"
I20260805 22:53:11.922545 4494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/wal/instance:
uuid: "037d3027adb3427d9953543c7c742bc9"
format_stamp: "Formatted at 2026-08-05 22:53:11 on dist-test-slave-7lbf"
server_key: "e59c60463836cc951fc9624233d98301"
server_key_iv: "4359d3043a97ab0a6861d026143e4559"
server_key_version: "encryptionkey@0"
I20260805 22:53:11.928632 4494 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260805 22:53:11.932655 4510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:53:11.934453 4494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260805 22:53:11.934657 4494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/wal
uuid: "037d3027adb3427d9953543c7c742bc9"
format_stamp: "Formatted at 2026-08-05 22:53:11 on dist-test-slave-7lbf"
server_key: "e59c60463836cc951fc9624233d98301"
server_key_iv: "4359d3043a97ab0a6861d026143e4559"
server_key_version: "encryptionkey@0"
I20260805 22:53:11.934900 4494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:53:11.955753 4494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:53:11.960414 4494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:53:11.960706 4494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:53:11.980540 4494 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.62:45061
I20260805 22:53:11.980547 4584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.62:45061 every 8 connection(s)
I20260805 22:53:11.982204 4494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/data/info.pb
I20260805 22:53:11.986637 4585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:53:11.988917 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4494
I20260805 22:53:11.989125 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-0/wal/instance
I20260805 22:53:11.989729 4208 external_mini_cluster.cc:1428] Setting key cfb64a6c121ce6bf35e3486819f3a92b
I20260805 22:53:11.996246 4585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:53:12 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970391, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:53:12.029210 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.000593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.4.28.62:45061)
0805 22:53:12.007339 (+ 6746us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:12.007348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:12.007415 (+ 67us) server_negotiation.cc:407] Connection header received
0805 22:53:12.009745 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:12.009753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:12.010062 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:12.010250 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:12.012005 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.013300 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.015510 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.016254 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.016438 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:12.020804 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:12.020815 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:12.020831 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:12.020872 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:12.024201 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:12.025304 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:12.025315 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:12.025319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:12.025598 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:12.026122 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:12.026126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:12.026128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:12.026718 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:12.026886 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:12.027657 (+ 771us) server_negotiation.cc:299] Negotiation successful
0805 22:53:12.028022 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":6496,"thread_start_us":6232,"threads_started":1}
I20260805 22:53:12.029291 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.000592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45061 (local address 127.0.0.1:56630)
0805 22:53:12.006287 (+ 5695us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.006353 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:12.009169 (+ 2816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:12.010362 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:12.010370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:12.011228 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:12.011742 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:12.011771 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.013568 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.013574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:12.015277 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:12.015300 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.016580 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.016592 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:12.016818 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:12.017620 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:12.017652 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:12.020510 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:12.024415 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:12.024433 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:12.024449 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:12.025100 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:12.025732 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:12.025738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:12.025742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:12.026000 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:12.026900 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:12.026909 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:12.027216 (+ 307us) client_negotiation.cc:769] Sending connection context
0805 22:53:12.027518 (+ 302us) client_negotiation.cc:240] Negotiation successful
0805 22:53:12.028216 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5331,"mutex_wait_us":112,"thread_start_us":5031,"threads_started":1}
WARNING: no policy specified for kudu/127.4.28.61@KRBTEST.COM; defaulting to no policy
I20260805 22:53:12.032537 4585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.61@KRBTEST.COM" created.
I20260805 22:53:12.033913 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.033440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48762)
0805 22:53:12.033655 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.033774 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260805 22:53:12.034297 4515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
I20260805 22:53:12.036167 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.035778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48772)
0805 22:53:12.035944 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.036025 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260805 22:53:12.036384 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40367: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab.
Entry for principal kudu/127.4.28.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab.
I20260805 22:53:12.075237 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.61:40367
--webserver_interface=127.4.28.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab
--principal=kudu/127.4.28.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45061,127.4.28.61:40367,127.4.28.60:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:53:12.085721 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } attempt: 1
I20260805 22:53:12.087086 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.086559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48786)
0805 22:53:12.086787 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.086906 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260805 22:53:12.088231 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.087806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48800)
0805 22:53:12.087987 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.088071 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260805 22:53:12.088466 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40367: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
I20260805 22:53:12.156755 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } attempt: 2
I20260805 22:53:12.158129 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.157623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48806)
0805 22:53:12.157867 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.157982 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260805 22:53:12.159205 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.158844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48812)
0805 22:53:12.158967 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.159060 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260805 22:53:12.159421 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40367: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
I20260805 22:53:12.250741 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } attempt: 3
I20260805 22:53:12.252125 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.251582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48820)
0805 22:53:12.251859 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.251962 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260805 22:53:12.253372 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.252966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48826)
0805 22:53:12.253115 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.253211 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260805 22:53:12.253638 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40367: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
I20260805 22:53:12.396965 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } attempt: 4
I20260805 22:53:12.398418 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.397821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48828)
0805 22:53:12.398177 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.398274 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260805 22:53:12.399535 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.399192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48844)
0805 22:53:12.399323 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.399398 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260805 22:53:12.399767 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.61:40367: Network error: Client connection negotiation failed: client connection to 127.4.28.61:40367: connect: Connection refused (error 111)
W20260805 22:53:12.458983 4592 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 22:53:12.459411 4592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:53:12.459542 4592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:53:12.459671 4592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:53:12.467679 4592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:53:12.467803 4592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:53:12.467883 4592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:53:12.467969 4592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:53:12.468066 4592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:53:12.479137 4592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45061,127.4.28.61:40367,127.4.28.60:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.61.keytab
--principal=kudu/127.4.28.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.61:40367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.28.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:53:12.480680 4592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:53:12.482316 4592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:53:12.495999 4601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:12.496060 4608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:12.496037 4603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:53:12.496757 4592 server_base.cc:1034] running on GCE node
I20260805 22:53:12.497769 4592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:53:12.499596 4592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:53:12.500875 4592 hybrid_clock.cc:648] HybridClock initialized: now 1785970392500769 us; error 71 us; skew 500 ppm
I20260805 22:53:12.501283 4592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:53:12 dist-test-slave-7lbf krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970392, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:53:12.504875 4592 init.cc:377] Logged in from keytab as kudu/127.4.28.61@KRBTEST.COM (short username kudu)
I20260805 22:53:12.506609 4592 webserver.cc:460] Webserver started at http://127.4.28.61:37995/ using document root <none> and password file <none>
I20260805 22:53:12.507397 4592 fs_manager.cc:359] Metadata directory not provided
I20260805 22:53:12.507517 4592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:53:12.507854 4592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:53:12.511307 4592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/data/instance:
uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa"
format_stamp: "Formatted at 2026-08-05 22:53:12 on dist-test-slave-7lbf"
server_key: "6e37f924f7f3c37b864af192a5183d4d"
server_key_iv: "105c940164726551099bbeb18a488ce6"
server_key_version: "encryptionkey@0"
I20260805 22:53:12.512229 4592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/wal/instance:
uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa"
format_stamp: "Formatted at 2026-08-05 22:53:12 on dist-test-slave-7lbf"
server_key: "6e37f924f7f3c37b864af192a5183d4d"
server_key_iv: "105c940164726551099bbeb18a488ce6"
server_key_version: "encryptionkey@0"
I20260805 22:53:12.518502 4592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260805 22:53:12.522857 4617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:53:12.524466 4592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 22:53:12.524688 4592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/wal
uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa"
format_stamp: "Formatted at 2026-08-05 22:53:12 on dist-test-slave-7lbf"
server_key: "6e37f924f7f3c37b864af192a5183d4d"
server_key_iv: "105c940164726551099bbeb18a488ce6"
server_key_version: "encryptionkey@0"
I20260805 22:53:12.524989 4592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:53:12.545799 4592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:53:12.550598 4592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:53:12.550907 4592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:53:12.569990 4592 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.61:40367
I20260805 22:53:12.570051 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.61:40367 every 8 connection(s)
I20260805 22:53:12.571502 4592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/data/info.pb
I20260805 22:53:12.575507 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4592
I20260805 22:53:12.575742 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-1/wal/instance
I20260805 22:53:12.575807 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:53:12.576416 4208 external_mini_cluster.cc:1428] Setting key 441dd30eddd9e951ac60dbb88f321767
I20260805 22:53:12.585284 4687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:53:12 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970392, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:53:12.611435 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.588931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.4.28.62:45061)
0805 22:53:12.591783 (+ 2852us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:12.591803 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:12.591832 (+ 29us) server_negotiation.cc:407] Connection header received
0805 22:53:12.593603 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:12.593611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:12.593712 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:12.593834 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:12.596351 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.597137 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.599641 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.600089 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.600207 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:12.604856 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:12.604863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:12.604866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:12.604960 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:12.607726 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:12.608651 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:12.608656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:12.608659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:12.608802 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:12.609276 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:12.609281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:12.609283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:12.609671 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:12.609823 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:12.610828 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0805 22:53:12.611115 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2710,"thread_start_us":2563,"threads_started":1}
I20260805 22:53:12.612785 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.589639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45061 (local address 127.0.0.1:56644)
0805 22:53:12.590940 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.591004 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:12.593175 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:12.594184 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:12.594199 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:12.595084 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:12.596038 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:12.596070 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.597373 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.597379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:12.599411 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:12.599434 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.600393 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.600401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:12.600744 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:12.601628 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:12.601666 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:12.604538 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:12.607931 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:12.607942 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:12.607960 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:12.608489 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:12.608925 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:12.608931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:12.608934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:12.609139 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:12.609823 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:12.609833 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:12.610241 (+ 408us) client_negotiation.cc:769] Sending connection context
0805 22:53:12.610639 (+ 398us) client_negotiation.cc:240] Negotiation successful
0805 22:53:12.611171 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":666,"threads_started":1}
WARNING: no policy specified for kudu/127.4.28.60@KRBTEST.COM; defaulting to no policy
I20260805 22:53:12.615010 4687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.28.60@KRBTEST.COM" created.
Aug 05 22:53:12 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970392, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
I20260805 22:53:12.631660 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.615963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48858)
0805 22:53:12.616208 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.616227 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:12.616331 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:12.617316 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:12.617320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:12.617837 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:12.618009 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:12.618019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.619924 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.619935 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:12.620817 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:12.620830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.622082 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.622087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:12.622240 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:12.622909 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:12.622929 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:12.625254 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:12.628722 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:12.628727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:12.628732 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:12.629182 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:12.629600 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:12.629605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:12.629609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:12.629782 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:12.630566 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:12.630571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:12.630733 (+ 162us) client_negotiation.cc:769] Sending connection context
0805 22:53:12.630882 (+ 149us) client_negotiation.cc:240] Negotiation successful
0805 22:53:12.631225 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260805 22:53:12.631700 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.616015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.4.28.61:40367)
0805 22:53:12.616796 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:12.616803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:12.616877 (+ 74us) server_negotiation.cc:407] Connection header received
0805 22:53:12.616931 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:12.616937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:12.617157 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:12.617293 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:12.618195 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.619679 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.620994 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.621921 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.622052 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:12.625573 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:12.625592 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:12.625613 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:12.625650 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:12.628530 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:12.629328 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:12.629336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:12.629338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:12.629468 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:12.629990 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:12.629994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:12.629996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:12.630395 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:12.630608 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:12.631087 (+ 479us) server_negotiation.cc:299] Negotiation successful
0805 22:53:12.631326 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":464,"threads_started":1}
I20260805 22:53:12.634603 4687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:53:12.635917 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.635407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45180)
0805 22:53:12.635618 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.635753 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260805 22:53:12.636287 4618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
I20260805 22:53:12.638212 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.637775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45186)
0805 22:53:12.637955 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.638056 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260805 22:53:12.638466 4687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.28.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab.
Entry for principal kudu/127.4.28.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab.
I20260805 22:53:12.662177 4208 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
/tmp/dist-test-taskW_LE9n/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.60:40133
--webserver_interface=127.4.28.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab
--principal=kudu/127.4.28.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.28.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.28.62:45061,127.4.28.61:40367,127.4.28.60:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 22:53:12.687999 4687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 1
I20260805 22:53:12.691771 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.691148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45198)
0805 22:53:12.691420 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.691559 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260805 22:53:12.693158 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.692655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45214)
0805 22:53:12.692866 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.692973 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260805 22:53:12.693507 4687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
I20260805 22:53:12.699016 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } attempt: 5
Aug 05 22:53:12 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970391, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
I20260805 22:53:12.713343 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.699876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.4.28.61:40367)
0805 22:53:12.700064 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:12.700070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:12.700095 (+ 25us) server_negotiation.cc:407] Connection header received
0805 22:53:12.700211 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:12.700217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:12.700339 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:12.700431 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:12.701379 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.702161 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.703425 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.703886 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.704008 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:12.707566 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:12.707572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:12.707575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:12.707615 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:12.710276 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:12.711029 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:12.711034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:12.711036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:12.711178 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:12.711726 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:12.711733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:12.711736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:12.712118 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:12.712277 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:12.712729 (+ 452us) server_negotiation.cc:299] Negotiation successful
0805 22:53:12.712975 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 22:53:12.713351 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.699709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48860)
0805 22:53:12.699918 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.699944 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:12.700089 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:12.700504 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:12.700510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:12.700993 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:12.701185 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:12.701196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.702348 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.702354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:12.703251 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:12.703264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:12.704058 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:12.704064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:12.704268 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:12.704888 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:12.704908 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:12.707306 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:12.710452 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:12.710459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:12.710464 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:12.710888 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:12.711305 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:12.711311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:12.711314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:12.711461 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:12.712258 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:12.712264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:12.712461 (+ 197us) client_negotiation.cc:769] Sending connection context
0805 22:53:12.712594 (+ 133us) client_negotiation.cc:240] Negotiation successful
0805 22:53:12.712963 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260805 22:53:12.714471 4585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
I20260805 22:53:12.715803 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.715345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45224)
0805 22:53:12.715546 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.715637 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260805 22:53:12.716835 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.716479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45232)
0805 22:53:12.716634 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.716723 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260805 22:53:12.717041 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
I20260805 22:53:12.761832 4687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 2
I20260805 22:53:12.763195 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.762641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45242)
0805 22:53:12.762894 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.763021 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260805 22:53:12.764350 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.763942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45244)
0805 22:53:12.764112 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.764198 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260805 22:53:12.764619 4687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
I20260805 22:53:12.768261 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 1
I20260805 22:53:12.769407 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.768907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45252)
0805 22:53:12.769137 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.769242 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260805 22:53:12.770450 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.770105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45264)
0805 22:53:12.770262 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.770335 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260805 22:53:12.770680 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
I20260805 22:53:12.838913 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 2
I20260805 22:53:12.840368 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.839692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45268)
0805 22:53:12.840000 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.840127 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260805 22:53:12.841657 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.841150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45274)
0805 22:53:12.841348 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.841455 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260805 22:53:12.841935 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
I20260805 22:53:12.855890 4687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 3
I20260805 22:53:12.857178 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.856625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45282)
0805 22:53:12.856864 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.856960 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260805 22:53:12.858592 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.858097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45288)
0805 22:53:12.858357 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.858451 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260805 22:53:12.858844 4687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
I20260805 22:53:12.948199 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 3
I20260805 22:53:12.949680 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.949018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45296)
0805 22:53:12.949278 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.949420 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260805 22:53:12.951079 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:12.950565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45298)
0805 22:53:12.950809 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:12.950917 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260805 22:53:12.951345 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
I20260805 22:53:13.002060 4687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 4
I20260805 22:53:13.003386 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.002824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45312)
0805 22:53:13.003107 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:13.003206 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260805 22:53:13.004527 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.004164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45316)
0805 22:53:13.004304 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:13.004385 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260805 22:53:13.004779 4687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
W20260805 22:53:13.051671 4697 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 22:53:13.052070 4697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 22:53:13.052184 4697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 22:53:13.052320 4697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 22:53:13.059988 4697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 22:53:13.060122 4697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 22:53:13.060211 4697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 22:53:13.060286 4697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 22:53:13.060375 4697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 22:53:13.070967 4697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.28.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.28.62:45061,127.4.28.61:40367,127.4.28.60:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/krb5kdc/kudu_127.4.28.60.keytab
--principal=kudu/127.4.28.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.28.60:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.28.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 05 Aug 2026 21:57:31 UTC on ccycloud.quasar-jetobj.root.comops.site
build id 4569245
ASAN enabled
I20260805 22:53:13.072507 4697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:53:13.074038 4697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:53:13.087667 4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:13.087626 4703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:53:13.087533 4709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:53:13.088469 4697 server_base.cc:1034] running on GCE node
I20260805 22:53:13.089398 4697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 22:53:13.091090 4697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 22:53:13.092366 4697 hybrid_clock.cc:648] HybridClock initialized: now 1785970393092220 us; error 128 us; skew 500 ppm
I20260805 22:53:13.092770 4697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 05 22:53:13 dist-test-slave-7lbf krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 22:53:13.096804 4697 init.cc:377] Logged in from keytab as kudu/127.4.28.60@KRBTEST.COM (short username kudu)
I20260805 22:53:13.098883 4697 webserver.cc:460] Webserver started at http://127.4.28.60:35931/ using document root <none> and password file <none>
I20260805 22:53:13.099733 4697 fs_manager.cc:359] Metadata directory not provided
I20260805 22:53:13.099865 4697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:53:13.100235 4697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:53:13.103754 4697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/data/instance:
uuid: "593933195e2b435ab59bf8dccad8e354"
format_stamp: "Formatted at 2026-08-05 22:53:13 on dist-test-slave-7lbf"
server_key: "2b30bb32c61a933a9aee22076c8bcfed"
server_key_iv: "7fd6664b39d66432d2423250f5d463c7"
server_key_version: "encryptionkey@0"
I20260805 22:53:13.104709 4697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/wal/instance:
uuid: "593933195e2b435ab59bf8dccad8e354"
format_stamp: "Formatted at 2026-08-05 22:53:13 on dist-test-slave-7lbf"
server_key: "2b30bb32c61a933a9aee22076c8bcfed"
server_key_iv: "7fd6664b39d66432d2423250f5d463c7"
server_key_version: "encryptionkey@0"
I20260805 22:53:13.111050 4697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260805 22:53:13.115494 4718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:53:13.117204 4697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260805 22:53:13.117429 4697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/data,/tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/wal
uuid: "593933195e2b435ab59bf8dccad8e354"
format_stamp: "Formatted at 2026-08-05 22:53:13 on dist-test-slave-7lbf"
server_key: "2b30bb32c61a933a9aee22076c8bcfed"
server_key_iv: "7fd6664b39d66432d2423250f5d463c7"
server_key_version: "encryptionkey@0"
I20260805 22:53:13.117749 4697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:53:13.128608 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 4
I20260805 22:53:13.129937 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.129367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45318)
0805 22:53:13.129658 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:13.129764 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260805 22:53:13.131095 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.130671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45332)
0805 22:53:13.130842 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:13.130936 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260805 22:53:13.131336 4585 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.28.60:40133: Network error: Client connection negotiation failed: client connection to 127.4.28.60:40133: connect: Connection refused (error 111)
I20260805 22:53:13.143061 4697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:53:13.148274 4697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:53:13.148573 4697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:53:13.167764 4697 rpc_server.cc:307] RPC server started. Bound to: 127.4.28.60:40133
I20260805 22:53:13.167801 4799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.28.60:40133 every 8 connection(s)
I20260805 22:53:13.169363 4697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/data/info.pb
I20260805 22:53:13.173449 4801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:53:13.175269 4208 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu as pid 4697
I20260805 22:53:13.175477 4208 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskW_LE9n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785970002413767-4208-0/minicluster-data/master-2/wal/instance
I20260805 22:53:13.176047 4208 external_mini_cluster.cc:1428] Setting key 011a9118ec30b910b0c4082d46a1e5c7
I20260805 22:53:13.183945 4801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:53:13 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
Aug 05 22:53:13 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:53:13.200906 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.178225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.4.28.60:40133)
0805 22:53:13.179186 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.179199 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.179251 (+ 52us) server_negotiation.cc:407] Connection header received
0805 22:53:13.180984 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.181017 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.181559 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.181979 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.184137 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.185674 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.187396 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.188528 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.188699 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.193033 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.193042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.193057 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.193107 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.196366 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.197169 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.197177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.197185 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.197354 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.197857 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.197861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.197863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.198456 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.198760 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.199286 (+ 526us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.199756 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":418,"threads_started":1}
I20260805 22:53:13.202971 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.187102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.4.28.62:45061)
0805 22:53:13.187706 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.187713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.188257 (+ 544us) server_negotiation.cc:407] Connection header received
0805 22:53:13.188501 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.188506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.188594 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.188710 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.190209 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.190981 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.192649 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.193087 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.193210 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.196930 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.196936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.196939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.196979 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.199770 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.200548 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.200553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.200555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.200704 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.201284 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.201290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.201304 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.201746 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.201924 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.202444 (+ 520us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.202658 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":269,"threads_started":1}
I20260805 22:53:13.203063 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.187081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.62:45061 (local address 127.0.0.1:56658)
0805 22:53:13.188094 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:13.188142 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:13.188329 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:13.188888 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:13.188896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:13.189744 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:13.190034 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.190045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.191155 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.191159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.192458 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.192477 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.193311 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.193319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.193545 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.194235 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:13.194254 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:13.196690 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:13.199908 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.199922 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.199931 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.200402 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.200840 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.200845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.200849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.200993 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.201894 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:13.201900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:13.202104 (+ 204us) client_negotiation.cc:769] Sending connection context
0805 22:53:13.202289 (+ 185us) client_negotiation.cc:240] Negotiation successful
0805 22:53:13.202626 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":547,"threads_started":1}
I20260805 22:53:13.204890 4801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:53:13 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
Aug 05 22:53:13 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.62@KRBTEST.COM
I20260805 22:53:13.221182 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.205627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.61:40367 (local address 127.0.0.1:48870)
0805 22:53:13.205815 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:13.205835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:13.205936 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:13.206404 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:13.206409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:13.206885 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:13.207047 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.207056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.208368 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.208374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.209818 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.209837 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.210841 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.210848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.211058 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.211816 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:13.211842 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:13.214702 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:13.218439 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.218445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.218449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.218923 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.219391 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.219396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.219401 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.219567 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.220155 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:13.220160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:13.220334 (+ 174us) client_negotiation.cc:769] Sending connection context
0805 22:53:13.220460 (+ 126us) client_negotiation.cc:240] Negotiation successful
0805 22:53:13.220794 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260805 22:53:13.221217 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.205963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.4.28.61:40367)
0805 22:53:13.206127 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.206131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.206147 (+ 16us) server_negotiation.cc:407] Connection header received
0805 22:53:13.206194 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.206198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.206256 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.206341 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.207205 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.208201 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.209984 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.210655 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.210863 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.215101 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.215108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.215112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.215158 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.218232 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.219084 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.219089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.219092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.219253 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.219697 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.219700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.219702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.220013 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.220173 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.220614 (+ 441us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.220843 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 22:53:13.221424 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.204525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.4.28.62:45061)
0805 22:53:13.204695 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.204700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.204776 (+ 76us) server_negotiation.cc:407] Connection header received
0805 22:53:13.204926 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.204931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.205044 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.205147 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.206902 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.208019 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.209984 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.210655 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.210798 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.215861 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.215868 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.215871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.215912 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.218572 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.219367 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.219372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.219374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.219520 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.219942 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.219946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.219948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.220254 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.220545 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.220871 (+ 326us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.221121 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 22:53:13.222978 4801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 22:53:13 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.60@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
Aug 05 22:53:13 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.28.61@KRBTEST.COM
I20260805 22:53:13.238267 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.224046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.4.28.60:40133)
0805 22:53:13.224211 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.224216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.224236 (+ 20us) server_negotiation.cc:407] Connection header received
0805 22:53:13.224291 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.224296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.224403 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.224502 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.225386 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.226414 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.227942 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.228635 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.228777 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.232798 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.232805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.232810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.232859 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.235609 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.236292 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.236298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.236301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.236432 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.236810 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.236814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.236816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.237113 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.237248 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.237628 (+ 380us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.237875 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260805 22:53:13.238402 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.223864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45344)
0805 22:53:13.224039 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:13.224056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:13.224151 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:13.224548 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:13.224553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:13.225033 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:13.225216 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.225226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.226554 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.226560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.227761 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.227775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.228806 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.228811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.229063 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.229871 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:13.229894 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:13.232454 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:13.235772 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.235778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.235783 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.236169 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.236560 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.236566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.236570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.236707 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.237227 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:13.237231 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:13.237373 (+ 142us) client_negotiation.cc:769] Sending connection context
0805 22:53:13.237495 (+ 122us) client_negotiation.cc:240] Negotiation successful
0805 22:53:13.238046 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":155}
I20260805 22:53:13.241096 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.223480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.4.28.61:40367)
0805 22:53:13.223687 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.223692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.224324 (+ 632us) server_negotiation.cc:407] Connection header received
0805 22:53:13.224493 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.224498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.224576 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.224670 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.226237 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.227247 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.228982 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.229672 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.229811 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.234918 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.234925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.234928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.234972 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.237603 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.238428 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.238434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.238437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.238674 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.239245 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.239251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.239254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.239690 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.240042 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.240436 (+ 394us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.240747 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260805 22:53:13.244807 4801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354: Bootstrap starting.
I20260805 22:53:13.246739 4208 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 22:53:13.248803 4801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354: Neither blocks nor log segments found. Creating new log.
I20260805 22:53:13.250120 4801 log.cc:824] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354: Log is configured to *not* fsync() on all Append() calls
I20260805 22:53:13.249490 4547 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "037d3027adb3427d9953543c7c742bc9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56664
I20260805 22:53:13.253298 4801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354: No bootstrap required, opened a new log
I20260805 22:53:13.261039 4801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } }
I20260805 22:53:13.261462 4801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593933195e2b435ab59bf8dccad8e354, State: Initialized, Role: FOLLOWER
I20260805 22:53:13.262023 4801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } }
I20260805 22:53:13.263216 4811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.263983 4811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:13.264535 4801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [sys.catalog]: configured and running, proceeding with master startup.
W20260805 22:53:13.275241 4824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 22:53:13.275435 4824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 22:53:13.272897 4697 master_runner.cc:186] Error getting master registration for 127.4.28.62:45061: OK, instance_id { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" instance_seqno: 1785970391974912 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 22:53:13.283339 4697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260805 22:53:13.304009 4687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 5
Aug 05 22:53:13 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970392, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.61@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
I20260805 22:53:13.320364 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.304764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45356)
0805 22:53:13.305019 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:13.305041 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:13.305184 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:13.305588 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:13.305594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:13.306034 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:13.306215 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.306226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.307354 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.307359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.308295 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.308311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.309125 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.309131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.309348 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.310096 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:13.310123 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:13.313134 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:13.317274 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.317288 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.317293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.317850 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.318390 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.318394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.318397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.318568 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.319251 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:13.319256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:13.319432 (+ 176us) client_negotiation.cc:769] Sending connection context
0805 22:53:13.319563 (+ 131us) client_negotiation.cc:240] Negotiation successful
0805 22:53:13.319934 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260805 22:53:13.320382 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.304898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.4.28.60:40133)
0805 22:53:13.305159 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.305164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.305188 (+ 24us) server_negotiation.cc:407] Connection header received
0805 22:53:13.305339 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.305344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.305418 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.305504 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.306399 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.307179 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.308465 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.308955 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.309096 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.313452 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.313459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.313462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.313535 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.317067 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.318041 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.318048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.318053 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.318242 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.318714 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.318721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.318725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.319105 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.319266 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.319752 (+ 486us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.320004 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260805 22:53:13.327282 4687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa: Bootstrap starting.
I20260805 22:53:13.332099 4687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa: Neither blocks nor log segments found. Creating new log.
I20260805 22:53:13.333702 4687 log.cc:824] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa: Log is configured to *not* fsync() on all Append() calls
I20260805 22:53:13.337878 4687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa: No bootstrap required, opened a new log
I20260805 22:53:13.349756 4687 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } }
I20260805 22:53:13.350306 4687 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ecccc1d7364cedba16f1ad07d9b6aa, State: Initialized, Role: FOLLOWER
I20260805 22:53:13.350961 4687 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } }
I20260805 22:53:13.352229 4828 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.352695 4828 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:13.353829 4687 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [sys.catalog]: configured and running, proceeding with master startup.
W20260805 22:53:13.366950 4845 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 22:53:13.367272 4845 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 22:53:13.362411 4592 master_runner.cc:186] Error getting master registration for 127.4.28.62:45061: OK, instance_id { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" instance_seqno: 1785970391974912 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 22:53:13.376003 4592 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260805 22:53:13.401144 4725 tablet.cc:2369] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:53:13.408564 4585 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } attempt: 5
Aug 05 22:53:13 dist-test-slave-7lbf krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785970391, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.28.62@KRBTEST.COM for kudu/127.4.28.60@KRBTEST.COM
I20260805 22:53:13.429847 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.409630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.4.28.60:40133)
0805 22:53:13.409855 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.409862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.409973 (+ 111us) server_negotiation.cc:407] Connection header received
0805 22:53:13.410200 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.410207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.410302 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.410422 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.411804 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.413019 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.415013 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.415798 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.415981 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.420828 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.420837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.420843 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.420900 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.425154 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.426303 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.426315 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.426319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.426552 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.427219 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.427225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.427229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.427667 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.427837 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.429075 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.429370 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260805 22:53:13.430065 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.409464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.28.60:40133 (local address 127.0.0.1:45372)
0805 22:53:13.409852 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0805 22:53:13.409875 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 22:53:13.410046 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 22:53:13.410719 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 22:53:13.410726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 22:53:13.411309 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0805 22:53:13.411576 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.411591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.413226 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.413233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.414803 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 22:53:13.414826 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.416029 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.416037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 22:53:13.416281 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.417166 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 22:53:13.417195 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 22:53:13.420524 (+ 3329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 22:53:13.425382 (+ 4858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.425390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.425395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.426089 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.426751 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 22:53:13.426758 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 22:53:13.426762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 22:53:13.427041 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 22:53:13.427918 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 22:53:13.427924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 22:53:13.428116 (+ 192us) client_negotiation.cc:769] Sending connection context
0805 22:53:13.428286 (+ 170us) client_negotiation.cc:240] Negotiation successful
0805 22:53:13.428648 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260805 22:53:13.437419 4585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9: Bootstrap starting.
I20260805 22:53:13.442202 4585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9: Neither blocks nor log segments found. Creating new log.
I20260805 22:53:13.443866 4585 log.cc:824] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9: Log is configured to *not* fsync() on all Append() calls
I20260805 22:53:13.448400 4585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9: No bootstrap required, opened a new log
I20260805 22:53:13.457381 4585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } }
I20260805 22:53:13.457933 4585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037d3027adb3427d9953543c7c742bc9, State: Initialized, Role: FOLLOWER
I20260805 22:53:13.458596 4585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } }
I20260805 22:53:13.460237 4849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.461099 4849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:13.461596 4585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:53:13.461669 4547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:53:13.461910 4547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:53:13.465754 4547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } }
W20260805 22:53:13.466944 4517 tablet.cc:2369] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:53:13.470036 4658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "037d3027adb3427d9953543c7c742bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa"
I20260805 22:53:13.470814 4658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:53:13.474983 4658 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 037d3027adb3427d9953543c7c742bc9 in term 1.
I20260805 22:53:13.476130 4515 leader_election.cc:302] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037d3027adb3427d9953543c7c742bc9, e3ecccc1d7364cedba16f1ad07d9b6aa; no voters:
I20260805 22:53:13.478400 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "037d3027adb3427d9953543c7c742bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "593933195e2b435ab59bf8dccad8e354"
I20260805 22:53:13.479031 4757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:53:13.483397 4757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 037d3027adb3427d9953543c7c742bc9 in term 1.
I20260805 22:53:13.484634 4547 leader_election.cc:288] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [CANDIDATE]: Term 1 election: Requested vote from peers e3ecccc1d7364cedba16f1ad07d9b6aa (127.4.28.61:40367), 593933195e2b435ab59bf8dccad8e354 (127.4.28.60:40133)
I20260805 22:53:13.485540 4849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:53:13.493618 4849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 037d3027adb3427d9953543c7c742bc9, State: Running, Role: LEADER
I20260805 22:53:13.494583 4849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } }
I20260805 22:53:13.503024 4856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037d3027adb3427d9953543c7c742bc9. Latest consensus state: current_term: 1 leader_uuid: "037d3027adb3427d9953543c7c742bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.503865 4856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:13.511088 4866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:53:13.525236 4866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:53:13.545387 4886 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260805 22:53:13.552807 4757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [term 1 FOLLOWER]: Refusing update from remote peer 037d3027adb3427d9953543c7c742bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:53:13.553232 4656 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [term 1 FOLLOWER]: Refusing update from remote peer 037d3027adb3427d9953543c7c742bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:53:13.554044 4628 tablet.cc:2369] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:53:13.554628 4856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:53:13.555512 4856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:53:13.578817 4828 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037d3027adb3427d9953543c7c742bc9. Latest consensus state: current_term: 1 leader_uuid: "037d3027adb3427d9953543c7c742bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.579180 4828 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:13.592087 4811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037d3027adb3427d9953543c7c742bc9. Latest consensus state: current_term: 1 leader_uuid: "037d3027adb3427d9953543c7c742bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.593457 4811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:13.594064 4856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "037d3027adb3427d9953543c7c742bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.594364 4856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:13.595726 4889 mvcc.cc:205] Tried to move back new op lower bound from 7315334731975315456 to 7315334731762184192. Current Snapshot: MvccSnapshot[applied={T|T < 7315334731975315456}]
I20260805 22:53:13.598277 4828 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "037d3027adb3427d9953543c7c742bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.598564 4828 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:13.604357 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.530250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.4.28.62:45061)
0805 22:53:13.530493 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.530501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.546173 (+ 15672us) server_negotiation.cc:407] Connection header received
0805 22:53:13.547630 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.547636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.547733 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.547833 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.561283 (+ 13450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.562456 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.575588 (+ 13132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.576260 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.576401 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.595660 (+ 19259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.595670 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.595676 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.595766 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.599939 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.601113 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.601119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.601123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.601327 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.601950 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.601956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.601960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.602434 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.602834 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.603535 (+ 701us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.603823 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 22:53:13.608527 4856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "037d3027adb3427d9953543c7c742bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.608844 4856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9 [sys.catalog]: This master's current role is: LEADER
I20260805 22:53:13.610353 4866 catalog_manager.cc:1347] Generated new cluster ID: 8b117593626b404095655480aaeff9c6
I20260805 22:53:13.610453 4866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:53:13.611088 4891 mvcc.cc:205] Tried to move back new op lower bound from 7315334731975315456 to 7315334731762184192. Current Snapshot: MvccSnapshot[applied={T|T < 7315334731975315456}]
I20260805 22:53:13.613703 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.544999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.4.28.61:40367)
0805 22:53:13.545248 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.545255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.558432 (+ 13177us) server_negotiation.cc:407] Connection header received
0805 22:53:13.565419 (+ 6987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.565428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.565565 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.565650 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.579898 (+ 14248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.581112 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.588830 (+ 7718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.589576 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.589690 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.605703 (+ 16013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.605712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.605718 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.605778 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.609914 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.611025 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.611033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.611037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.611233 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.611788 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.611794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.611797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.612225 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.612463 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.613029 (+ 566us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.613294 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260805 22:53:13.614723 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:13.532975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.4.28.60:40133)
0805 22:53:13.533240 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.533248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.553457 (+ 20209us) server_negotiation.cc:407] Connection header received
0805 22:53:13.558592 (+ 5135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.558597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.558684 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.558755 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.574134 (+ 15379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.575279 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.584992 (+ 9713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:13.585883 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:13.586076 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:13.605636 (+ 19560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 22:53:13.605644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 22:53:13.605649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 22:53:13.605700 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0805 22:53:13.609985 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.611773 (+ 1788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.611780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.611785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.611971 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 22:53:13.612450 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 22:53:13.612456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 22:53:13.612459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 22:53:13.612853 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 22:53:13.613096 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:13.614067 (+ 971us) server_negotiation.cc:299] Negotiation successful
0805 22:53:13.614325 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260805 22:53:13.620151 4811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "037d3027adb3427d9953543c7c742bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d3027adb3427d9953543c7c742bc9" member_type: VOTER last_known_addr { host: "127.4.28.62" port: 45061 } } peers { permanent_uuid: "e3ecccc1d7364cedba16f1ad07d9b6aa" member_type: VOTER last_known_addr { host: "127.4.28.61" port: 40367 } } peers { permanent_uuid: "593933195e2b435ab59bf8dccad8e354" member_type: VOTER last_known_addr { host: "127.4.28.60" port: 40133 } } }
I20260805 22:53:13.620471 4811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 22:53:13.680765 4866 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:53:13.683190 4866 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:53:13.716286 4866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 037d3027adb3427d9953543c7c742bc9: Generated new TSK 0
I20260805 22:53:13.717757 4866 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 22:53:13.884526 4807 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:13.860718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.4.28.62:45061)
0805 22:53:13.860966 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.860986 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.881660 (+ 20674us) server_negotiation.cc:407] Connection header received
0805 22:53:13.882053 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.882060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.882165 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.882259 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.884228 (+ 1969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56688: BlockingRecv error: recv got EOF from 127.0.0.1:56688 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260805 22:53:13.888249 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:13.867432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.4.28.60:40133)
0805 22:53:13.867683 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.867691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.885493 (+ 17802us) server_negotiation.cc:407] Connection header received
0805 22:53:13.886546 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.886552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.886661 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.886729 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.887952 (+ 1223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45400: BlockingRecv error: recv got EOF from 127.0.0.1:45400 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260805 22:53:13.889263 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:13.881101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.4.28.61:40367)
0805 22:53:13.881358 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.881365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.886177 (+ 4812us) server_negotiation.cc:407] Connection header received
0805 22:53:13.886388 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.886394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.886536 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.886637 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.888954 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48894: BlockingRecv error: recv got EOF from 127.0.0.1:48894 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260805 22:53:13.896399 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:13.893918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.4.28.61:40367)
0805 22:53:13.894149 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.894157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.894714 (+ 557us) server_negotiation.cc:407] Connection header received
0805 22:53:13.894971 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.894983 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.895126 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.895230 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.896118 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48898: BlockingRecv error: recv got EOF from 127.0.0.1:48898 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260805 22:53:13.906841 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:13.904336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.4.28.61:40367)
0805 22:53:13.904589 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.904596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.905175 (+ 579us) server_negotiation.cc:407] Connection header received
0805 22:53:13.905399 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.905406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.905495 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.905634 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.906594 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48904: BlockingRecv error: recv got EOF from 127.0.0.1:48904 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260805 22:53:13.914506 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:13.912068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.4.28.61:40367)
0805 22:53:13.912311 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.912320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.912893 (+ 573us) server_negotiation.cc:407] Connection header received
0805 22:53:13.913108 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.913115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.913198 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.913292 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.914227 (+ 935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48910: BlockingRecv error: recv got EOF from 127.0.0.1:48910 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260805 22:53:13.930713 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:13.927964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.4.28.61:40367)
0805 22:53:13.928279 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.928286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.928972 (+ 686us) server_negotiation.cc:407] Connection header received
0805 22:53:13.929188 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.929193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.929289 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.929380 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.930388 (+ 1008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48924: BlockingRecv error: recv got EOF from 127.0.0.1:48924 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
W20260805 22:53:13.944950 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:13.942588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.4.28.61:40367)
0805 22:53:13.942851 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.942858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.943438 (+ 580us) server_negotiation.cc:407] Connection header received
0805 22:53:13.943658 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.943662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.943741 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.943842 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.944670 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48930: BlockingRecv error: recv got EOF from 127.0.0.1:48930 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260805 22:53:13.967500 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:13.964900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.4.28.61:40367)
0805 22:53:13.965142 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:13.965149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:13.965732 (+ 583us) server_negotiation.cc:407] Connection header received
0805 22:53:13.966114 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:13.966119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:13.966205 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:13.966295 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:13.967220 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48932: BlockingRecv error: recv got EOF from 127.0.0.1:48932 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260805 22:53:14.005694 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:14.003128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.4.28.61:40367)
0805 22:53:14.003375 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:14.003383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:14.004037 (+ 654us) server_negotiation.cc:407] Connection header received
0805 22:53:14.004266 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:14.004270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:14.004356 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:14.004453 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:14.005339 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48940: BlockingRecv error: recv got EOF from 127.0.0.1:48940 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260805 22:53:14.075872 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:14.073316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.4.28.61:40367)
0805 22:53:14.073621 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:14.073630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:14.074240 (+ 610us) server_negotiation.cc:407] Connection header received
0805 22:53:14.074501 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:14.074506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:14.074596 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:14.074698 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:14.075581 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48956: BlockingRecv error: recv got EOF from 127.0.0.1:48956 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260805 22:53:14.210450 4693 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 22:53:14.208156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.4.28.61:40367)
0805 22:53:14.208405 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:14.208412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:14.208654 (+ 242us) server_negotiation.cc:407] Connection header received
0805 22:53:14.208935 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:14.208940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:14.209023 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:14.209121 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 22:53:14.210202 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48966: BlockingRecv error: recv got EOF from 127.0.0.1:48966 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260805 22:53:14.277841 4824 catalog_manager.cc:1259] Loaded cluster ID: 8b117593626b404095655480aaeff9c6
I20260805 22:53:14.277963 4824 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354: loading cluster ID for follower catalog manager: success
I20260805 22:53:14.280768 4824 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354: acquiring CA information for follower catalog manager: success
I20260805 22:53:14.283464 4824 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 593933195e2b435ab59bf8dccad8e354: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 22:53:14.369817 4845 catalog_manager.cc:1259] Loaded cluster ID: 8b117593626b404095655480aaeff9c6
I20260805 22:53:14.369931 4845 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa: loading cluster ID for follower catalog manager: success
I20260805 22:53:14.372663 4845 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa: acquiring CA information for follower catalog manager: success
I20260805 22:53:14.374819 4845 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e3ecccc1d7364cedba16f1ad07d9b6aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 22:53:14.476609 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:14.470502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.4.28.61:40367)
0805 22:53:14.470769 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:14.470776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:14.470979 (+ 203us) server_negotiation.cc:407] Connection header received
0805 22:53:14.471142 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:14.471150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:14.471334 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:14.471454 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:14.471841 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.472598 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.474184 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.474616 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.474722 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:14.475301 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:14.475880 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:14.475998 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:14.476203 (+ 205us) server_negotiation.cc:299] Negotiation successful
0805 22:53:14.476287 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260805 22:53:14.489347 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:14.483183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.4.28.60:40133)
0805 22:53:14.483647 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:14.483652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:14.483760 (+ 108us) server_negotiation.cc:407] Connection header received
0805 22:53:14.483962 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:14.483969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:14.484174 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:14.484375 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:14.484844 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.485616 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.487017 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.487419 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.487544 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:14.488101 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:14.488668 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:14.488823 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:14.488931 (+ 108us) server_negotiation.cc:299] Negotiation successful
0805 22:53:14.489021 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":221,"threads_started":1}
I20260805 22:53:14.504451 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:14.495575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.4.28.61:40367)
0805 22:53:14.496355 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:14.496362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:14.498335 (+ 1973us) server_negotiation.cc:407] Connection header received
0805 22:53:14.498565 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:14.498571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:14.498673 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:14.498749 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:14.499631 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.500619 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.502320 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.502769 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.502885 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:14.503680 (+ 795us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:14.503802 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:14.503882 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:14.504042 (+ 160us) server_negotiation.cc:299] Negotiation successful
0805 22:53:14.504132 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642}
I20260805 22:53:14.505694 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:14.495789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.4.28.60:40133)
0805 22:53:14.496517 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:14.496523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:14.498123 (+ 1600us) server_negotiation.cc:407] Connection header received
0805 22:53:14.498467 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:14.498473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:14.498556 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:14.498644 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:14.499350 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.500157 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.502963 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.503676 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.503794 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:14.504915 (+ 1121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:14.505055 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:14.505152 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:14.505337 (+ 185us) server_negotiation.cc:299] Negotiation successful
0805 22:53:14.505417 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597}
I20260805 22:53:14.506448 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 22:53:14.495575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.4.28.62:45061)
0805 22:53:14.496921 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0805 22:53:14.496927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 22:53:14.497598 (+ 671us) server_negotiation.cc:407] Connection header received
0805 22:53:14.498048 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 22:53:14.498054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 22:53:14.498271 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 22:53:14.498439 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 22:53:14.499255 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.500298 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.502964 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 22:53:14.503683 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 22:53:14.503814 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 22:53:14.504738 (+ 924us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 22:53:14.505220 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 22:53:14.505322 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0805 22:53:14.506060 (+ 738us) server_negotiation.cc:299] Negotiation successful
0805 22:53:14.506153 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":1074,"threads_started":1}
I20260805 22:53:14.511603 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4494
I20260805 22:53:14.512621 4575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:53:14.623468 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4494
I20260805 22:53:14.656749 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4592
I20260805 22:53:14.658063 4681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:53:14.769821 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4592
I20260805 22:53:14.797968 4208 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu pid 4697
I20260805 22:53:14.799186 4793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260805 22:53:14.908419 4208 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskW_LE9n/build/asan/bin/kudu with pid 4697
2026-08-05T22:53:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5071 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10154 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (392419 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST